Sep 17 18:00:46.190426 ip-10-0-137-104 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 17 18:00:46.190438 ip-10-0-137-104 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 17 18:00:46.190445 ip-10-0-137-104 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 17 18:00:46.190610 ip-10-0-137-104 systemd[1]: Failed to start Kubernetes Kubelet. Sep 17 18:00:56.352528 ip-10-0-137-104 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 17 18:00:56.352544 ip-10-0-137-104 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4acc98afed32449daa2faa3206926314 -- Sep 17 18:03:07.327154 ip-10-0-137-104 systemd[1]: Starting Kubernetes Kubelet... Sep 17 18:03:11.119076 ip-10-0-137-104 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:03:11.119076 ip-10-0-137-104 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 18:03:11.119076 ip-10-0-137-104 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:03:11.119076 ip-10-0-137-104 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 18:03:11.119076 ip-10-0-137-104 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 18:03:11.119835 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.119750 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 18:03:11.123965 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.123950 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:03:11.124004 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.123988 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:03:11.124004 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.123992 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:03:11.124004 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.123996 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:03:11.124004 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.123999 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:03:11.124004 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124002 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:03:11.124004 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124005 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124008 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124016 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124018 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124021 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124024 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124026 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124029 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124032 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124034 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124037 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124041 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124045 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124048 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124051 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124053 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124056 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124059 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124061 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124064 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:03:11.124145 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124066 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124069 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124071 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124074 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124076 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124079 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124081 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124083 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124086 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124089 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124091 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124094 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124096 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124099 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124103 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124105 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124108 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124111 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124113 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124116 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:03:11.124619 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124119 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124121 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124123 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124126 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124128 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124131 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124133 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124136 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124144 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124147 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124149 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124152 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124155 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124158 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124160 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124163 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124165 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124167 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124170 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124172 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:03:11.125128 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124175 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124177 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124180 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124182 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124185 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124187 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124191 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124194 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124196 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124199 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124202 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124204 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124207 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124209 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124212 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124214 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124217 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124219 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124223 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124227 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:03:11.125607 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124230 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124233 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124235 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124238 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124240 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124245 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124249 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124252 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124254 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124256 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124259 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124262 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.124264 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125652 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125659 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125663 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125666 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125669 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125672 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:03:11.126090 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125676 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125678 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125681 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125683 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125686 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125688 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125691 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125693 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125696 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125698 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125701 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125703 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125706 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125708 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125710 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125715 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125717 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125720 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125724 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125726 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:03:11.126544 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125729 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125731 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125734 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125737 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125739 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125742 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125744 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125747 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125749 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125752 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125754 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125757 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125760 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125763 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125765 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125768 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125770 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125773 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125775 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125778 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:03:11.127054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125781 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125784 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125787 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125789 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125792 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125794 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125797 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125802 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125804 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125807 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125810 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125812 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125815 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125818 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125820 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125823 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125825 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125828 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125830 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125833 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:03:11.127557 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125835 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125838 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125840 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125843 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125845 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125848 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125850 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125853 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125855 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125875 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125879 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125882 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125885 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125889 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125892 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125894 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125897 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125899 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125905 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:03:11.128066 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125909 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125913 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125915 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125918 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125921 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125923 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125926 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125929 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125933 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125936 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125939 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125941 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125944 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.125946 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126325 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126329 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126332 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126335 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126338 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:03:11.128529 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126342 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126344 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126347 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126350 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126352 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126355 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126358 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126360 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126363 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126365 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126368 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126371 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126374 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126376 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126379 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126381 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126384 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126387 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126389 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126392 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:03:11.129029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126394 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126397 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126399 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126402 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126404 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126407 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126409 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126412 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126414 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126417 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126420 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126422 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126425 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126428 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126431 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126433 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126436 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126438 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126441 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126443 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:03:11.129515 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126446 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126448 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126451 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126454 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126457 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126459 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126462 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126464 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126467 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126469 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126472 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126474 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126477 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126480 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126482 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126486 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126490 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126493 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126496 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:03:11.130011 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126499 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126503 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126506 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126510 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126513 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126516 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126520 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126524 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126527 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126530 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126533 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126535 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126538 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126540 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126542 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126545 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126547 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126550 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126553 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:03:11.130496 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126555 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126559 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126562 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126564 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126567 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126569 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126571 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126574 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126576 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126579 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126581 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126584 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126586 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126589 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126591 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.126594 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126665 2582 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126695 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126701 2582 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126705 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126709 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 18:03:11.130971 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126713 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126718 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126722 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126725 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126728 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126731 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126735 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126738 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126741 2582 flags.go:64] FLAG: --cgroup-root="" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126744 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126747 2582 flags.go:64] FLAG: --client-ca-file="" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126750 2582 flags.go:64] FLAG: --cloud-provider="external" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126753 2582 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126757 2582 flags.go:64] FLAG: --cluster-domain="" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126760 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126763 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126766 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126769 2582 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126773 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126776 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126779 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126782 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126785 2582 flags.go:64] FLAG: --contention-profiling="false" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126787 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 18:03:11.131475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126790 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126793 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126796 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126802 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126805 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126808 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126811 2582 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126814 2582 flags.go:64] FLAG: --enable-server="true" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126816 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126820 2582 flags.go:64] FLAG: --event-burst="100" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126824 2582 flags.go:64] FLAG: --event-qps="50" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126827 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126830 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126832 2582 flags.go:64] FLAG: --eviction-hard="" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126836 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126839 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126842 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126845 2582 flags.go:64] FLAG: --eviction-soft="" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126848 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126850 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126853 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126856 2582 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126872 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126875 2582 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126878 2582 flags.go:64] FLAG: --feature-gates="" Sep 17 18:03:11.132053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126881 2582 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126884 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126888 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126891 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126893 2582 flags.go:64] FLAG: --healthz-port="10248" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126896 2582 flags.go:64] FLAG: --help="false" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126899 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126902 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126905 2582 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126908 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126919 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126924 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126926 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126929 2582 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126932 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126935 2582 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126938 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126942 2582 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126945 2582 flags.go:64] FLAG: --kube-reserved="" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126948 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126951 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126954 2582 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126956 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126959 2582 flags.go:64] FLAG: --lock-file="" Sep 17 18:03:11.132682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126962 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126965 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126968 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126973 2582 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126976 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126979 2582 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126981 2582 flags.go:64] FLAG: --logging-format="text" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126984 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126987 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126990 2582 flags.go:64] FLAG: --manifest-url="" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126993 2582 flags.go:64] FLAG: --manifest-url-header="" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.126997 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127000 2582 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127004 2582 flags.go:64] FLAG: --max-pods="110" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127006 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127009 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127013 2582 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127015 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127020 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127023 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127026 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127032 2582 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127035 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127038 2582 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 18:03:11.133272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127041 2582 flags.go:64] FLAG: --pod-cidr="" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127044 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127050 2582 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127053 2582 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127057 2582 flags.go:64] FLAG: --pods-per-core="0" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127060 2582 flags.go:64] FLAG: --port="10250" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127063 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127066 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05eba4813d70d4ebc" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127069 2582 flags.go:64] FLAG: --qos-reserved="" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127072 2582 flags.go:64] FLAG: --read-only-port="10255" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127075 2582 flags.go:64] FLAG: --register-node="true" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127078 2582 flags.go:64] FLAG: --register-with-taints="" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127082 2582 flags.go:64] FLAG: --registry-burst="10" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127085 2582 flags.go:64] FLAG: --registry-qps="5" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127087 2582 flags.go:64] FLAG: --reserved-cpus="" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127090 2582 flags.go:64] FLAG: --reserved-memory="" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127094 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127097 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127100 2582 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127103 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127106 2582 flags.go:64] FLAG: --runonce="false" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127109 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127111 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127114 2582 flags.go:64] FLAG: --seccomp-default="false" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127117 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 18:03:11.133906 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127122 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127127 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127130 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127133 2582 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127136 2582 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127139 2582 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127142 2582 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127144 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127148 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127151 2582 flags.go:64] FLAG: --system-cgroups="" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127154 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127159 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127162 2582 flags.go:64] FLAG: --tls-cert-file="" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127165 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127168 2582 flags.go:64] FLAG: --tls-min-version="" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127171 2582 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127174 2582 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127177 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127180 2582 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127183 2582 flags.go:64] FLAG: --v="2" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127187 2582 flags.go:64] FLAG: --version="false" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127191 2582 flags.go:64] FLAG: --vmodule="" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127194 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.127197 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127308 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:03:11.134524 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127311 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127315 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127318 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127320 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127323 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127326 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127329 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127333 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127338 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127341 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127344 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127346 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127349 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127352 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127355 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127357 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127360 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127363 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127366 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127369 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:03:11.135558 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127371 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127374 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127377 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127379 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127383 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127386 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127390 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127393 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127395 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127398 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127401 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127403 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127406 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127408 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127419 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127421 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127424 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127426 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127429 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:03:11.136085 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127433 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127437 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127440 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127442 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127445 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127449 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127453 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127456 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127459 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127462 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127465 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127469 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127475 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127478 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127481 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127483 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127486 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127489 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127491 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:03:11.136550 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127494 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127496 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127499 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127501 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127504 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127506 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127509 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127511 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127514 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127516 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127518 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127521 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127524 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127528 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127532 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127535 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127537 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127540 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127542 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127545 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:03:11.137019 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127548 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127550 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127553 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127555 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127558 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127561 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127564 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127567 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127570 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127573 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127575 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127578 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127580 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127583 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127585 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127588 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127590 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127593 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127595 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:03:11.137510 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.127598 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.128488 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.136747 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.136763 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.136787 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.136793 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136848 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136854 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136858 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136878 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136882 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136885 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136888 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136892 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:03:11.137973 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136896 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136899 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136902 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136905 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136909 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136912 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136915 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136917 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136920 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136923 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136925 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136928 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136931 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136934 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136937 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136940 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136943 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136946 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136949 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136951 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136954 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:03:11.138384 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136957 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136960 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136963 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136965 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136968 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136971 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136973 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136976 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136979 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136981 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136984 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136986 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136989 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136991 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136994 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136997 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.136999 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137002 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137004 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:03:11.138932 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137007 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137010 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137013 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137017 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137020 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137023 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137026 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137029 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137032 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137034 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137037 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137040 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137042 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137047 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137049 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137052 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137055 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137057 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137060 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137062 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:03:11.139385 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137065 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137068 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137071 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137074 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137076 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137079 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137082 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137084 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137087 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137089 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137091 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137095 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137098 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137100 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137103 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137105 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137108 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137110 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137113 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137115 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:03:11.139920 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137118 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137120 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137123 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137125 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137128 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137131 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137134 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137137 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137140 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137142 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137145 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.137150 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137265 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137270 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 18:03:11.140401 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137274 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137279 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137282 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137284 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137287 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137289 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137292 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137295 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137298 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137300 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137304 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137307 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137309 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137312 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137314 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137317 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137319 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137322 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137325 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137327 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 18:03:11.140744 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137330 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137333 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137335 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137339 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137341 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137345 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137347 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137350 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137353 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137355 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137358 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137361 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137363 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137366 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137368 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137371 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137373 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137376 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137378 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137382 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 18:03:11.141259 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137385 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137388 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137391 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137394 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137397 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137400 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137403 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137406 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137409 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137412 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137415 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137418 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137420 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137423 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137425 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137428 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137431 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137434 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137436 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 18:03:11.141740 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137439 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137441 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137444 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137447 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137449 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137451 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137454 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137456 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137459 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137461 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137464 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137466 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137469 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137471 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137474 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137476 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137478 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137482 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137485 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137487 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 18:03:11.142206 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137490 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137492 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137495 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137497 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137501 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137504 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137507 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137510 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137515 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137518 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137521 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137524 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137526 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137529 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137532 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137535 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137538 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 18:03:11.142678 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:11.137540 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 18:03:11.143179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.137545 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 18:03:11.143179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.137666 2582 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 18:03:11.143179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.140415 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 18:03:11.143179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.141339 2582 server.go:1019] "Starting client certificate rotation" Sep 17 18:03:11.143179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.141434 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 18:03:11.143179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.141473 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 18:03:11.143793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.143779 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 18:03:11.144894 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.144878 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 18:03:11.149683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.149666 2582 log.go:25] "Validated CRI v1 runtime API" Sep 17 18:03:11.151074 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.151061 2582 log.go:25] "Validated CRI v1 image API" Sep 17 18:03:11.151915 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.151899 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 18:03:11.155099 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.155081 2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 92d1beea-687b-4a47-8421-18114717001f:/dev/nvme0n1p4 930263e6-7e74-4331-a8e8-acac8d71ff7a:/dev/nvme0n1p3] Sep 17 18:03:11.155166 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.155098 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 18:03:11.160922 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.160795 2582 manager.go:216] Machine: {Timestamp:2026-09-17 18:03:11.160323067 +0000 UTC m=+2.224267016 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099130 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c2c28da7c27bb97435bf8899102e SystemUUID:ec29c2c2-8da7-c27b-b974-35bf8899102e BootID:4acc98af-ed32-449d-aa2f-aa3206926314 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:fe:ce:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:fe:ce:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:5c:3d:75:91:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 18:03:11.160922 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.160909 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 18:03:11.161080 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.160982 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 18:03:11.162339 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.162314 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 18:03:11.162464 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.162338 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-104.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 18:03:11.163524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.163512 2582 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 18:03:11.163569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.163531 2582 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 18:03:11.163569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.163539 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 18:03:11.163569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.163550 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 18:03:11.164644 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.164635 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 18:03:11.167992 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.167978 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 17 18:03:11.168106 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.168097 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 18:03:11.168154 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.168130 2582 kubelet.go:486] "Attempting to sync node with API server" Sep 17 18:03:11.168154 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.168141 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 18:03:11.168214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.168155 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 18:03:11.168214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.168165 2582 kubelet.go:398] "Adding apiserver pod source" Sep 17 18:03:11.168214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.168177 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 18:03:11.168505 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.168493 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 18:03:11.168558 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.168511 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 18:03:11.175103 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.171598 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 18:03:11.175103 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.173082 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 18:03:11.175200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.175168 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 18:03:11.175234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.175210 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 18:03:11.176230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176216 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 18:03:11.176261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176239 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 18:03:11.176261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176247 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 18:03:11.176261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176252 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 18:03:11.176261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176260 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176265 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176278 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176285 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176291 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176297 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176309 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176315 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176342 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 18:03:11.176360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.176350 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 18:03:11.180178 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.180141 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-104.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 18:03:11.180214 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.180199 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 18:03:11.180297 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.180275 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 18:03:11.180471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.180453 2582 server.go:1268] "Started kubelet" Sep 17 18:03:11.180562 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.180544 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 18:03:11.180655 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.180610 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 18:03:11.180700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.180672 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 18:03:11.182003 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.181964 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 18:03:11.182072 ip-10-0-137-104 systemd[1]: Started Kubernetes Kubelet. Sep 17 18:03:11.182639 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.182404 2582 server.go:310] "Adding debug handlers to kubelet server" Sep 17 18:03:11.185232 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.185208 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 18:03:11.187457 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.187439 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 18:03:11.187457 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.187455 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 18:03:11.187840 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.187671 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:11.187991 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.187975 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 18:03:11.188061 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.187994 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 18:03:11.188116 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.187979 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 18:03:11.188166 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.188116 2582 reconstruct.go:97] "Volume reconstruction finished" Sep 17 18:03:11.188166 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.188127 2582 reconciler.go:29] "Reconciler: start to sync state" Sep 17 18:03:11.188375 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.188356 2582 factory.go:55] Registering systemd factory Sep 17 18:03:11.188464 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.188382 2582 factory.go:223] Registration of the systemd container factory successfully Sep 17 18:03:11.210016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.209995 2582 factory.go:153] Registering CRI-O factory Sep 17 18:03:11.210016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.210019 2582 factory.go:223] Registration of the crio container factory successfully Sep 17 18:03:11.210162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.210076 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 18:03:11.210162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.210103 2582 factory.go:103] Registering Raw factory Sep 17 18:03:11.210162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.210135 2582 manager.go:1196] Started watching for new ooms in manager Sep 17 18:03:11.210297 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.210186 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-104.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 18:03:11.210372 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.210301 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 18:03:11.212137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.212108 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rt44p" Sep 17 18:03:11.212515 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.212498 2582 manager.go:349] Starting recovery of all containers Sep 17 18:03:11.213245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.213218 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rt44p" Sep 17 18:03:11.223097 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.223081 2582 manager.go:354] Recovery completed Sep 17 18:03:11.227263 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.227241 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:03:11.228954 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.228940 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:03:11.229022 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.228967 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:03:11.229022 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.228982 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:03:11.229499 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.229487 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 18:03:11.229499 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.229492 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:03:11.229499 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.229499 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 18:03:11.229604 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.229517 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 17 18:03:11.230164 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.230150 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:03:11.230210 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.230176 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:03:11.230210 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.230192 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:03:11.230783 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.230770 2582 policy_none.go:49] "None policy: Start" Sep 17 18:03:11.230833 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.230787 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 18:03:11.230833 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.230797 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 18:03:11.231688 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.231677 2582 policy_none.go:47] "Start" Sep 17 18:03:11.274559 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.274434 2582 manager.go:337] "Starting Device Plugin manager" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.274591 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.274605 2582 server.go:87] "Starting device plugin registration server" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.274803 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.274820 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.274937 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.275015 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.275024 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.275599 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 18:03:11.286289 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.275646 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:11.337227 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.337193 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 18:03:11.338383 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.338367 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 18:03:11.338457 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.338424 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 18:03:11.338457 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.338450 2582 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 18:03:11.338558 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.338501 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 18:03:11.340893 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.340856 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:03:11.374994 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.374934 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:03:11.376327 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.376313 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:03:11.376376 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.376363 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:03:11.376409 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.376379 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:03:11.376409 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.376407 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.387937 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.387919 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.387997 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.387947 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-104.ec2.internal\": node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:11.407725 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.407704 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:11.439356 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.439326 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal"] Sep 17 18:03:11.439474 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.439394 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:03:11.440786 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.440772 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:03:11.440882 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.440799 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:03:11.440882 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.440810 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:03:11.441939 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.441927 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:03:11.442090 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.442073 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.442178 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.442108 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:03:11.442640 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.442623 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:03:11.442640 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.442635 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:03:11.442754 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.442649 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:03:11.442754 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.442661 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:03:11.442754 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.442671 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:03:11.442754 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.442681 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:03:11.443881 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.443851 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.443960 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.443888 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 18:03:11.444623 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.444606 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientMemory" Sep 17 18:03:11.444696 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.444637 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 18:03:11.444696 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.444661 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeHasSufficientPID" Sep 17 18:03:11.479555 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.479534 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-104.ec2.internal\" not found" node="ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.484064 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.484048 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-104.ec2.internal\" not found" node="ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.508430 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.508410 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:11.589288 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.589263 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70d28bf90c68089cd4d8c2ce3d2b7fc-config\") pod \"kube-apiserver-proxy-ip-10-0-137-104.ec2.internal\" (UID: \"a70d28bf90c68089cd4d8c2ce3d2b7fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.589394 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.589291 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a81d5fc529fcc24ca4d74829ae35e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal\" (UID: \"73a81d5fc529fcc24ca4d74829ae35e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.589394 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.589315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a81d5fc529fcc24ca4d74829ae35e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal\" (UID: \"73a81d5fc529fcc24ca4d74829ae35e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.609277 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.609248 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:11.690085 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.690064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70d28bf90c68089cd4d8c2ce3d2b7fc-config\") pod \"kube-apiserver-proxy-ip-10-0-137-104.ec2.internal\" (UID: \"a70d28bf90c68089cd4d8c2ce3d2b7fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.690157 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.690101 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a81d5fc529fcc24ca4d74829ae35e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal\" (UID: \"73a81d5fc529fcc24ca4d74829ae35e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.690157 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.690120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a81d5fc529fcc24ca4d74829ae35e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal\" (UID: \"73a81d5fc529fcc24ca4d74829ae35e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.690252 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.690204 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73a81d5fc529fcc24ca4d74829ae35e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal\" (UID: \"73a81d5fc529fcc24ca4d74829ae35e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.690293 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.690254 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a70d28bf90c68089cd4d8c2ce3d2b7fc-config\") pod \"kube-apiserver-proxy-ip-10-0-137-104.ec2.internal\" (UID: \"a70d28bf90c68089cd4d8c2ce3d2b7fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.690333 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.690300 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73a81d5fc529fcc24ca4d74829ae35e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal\" (UID: \"73a81d5fc529fcc24ca4d74829ae35e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.710191 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.710169 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:11.783370 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.783347 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.787564 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:11.787538 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" Sep 17 18:03:11.810287 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.810264 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:11.910771 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:11.910743 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:12.011331 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.011280 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:12.111759 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.111728 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-104.ec2.internal\" not found" Sep 17 18:03:12.140448 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.140423 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:03:12.140959 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.140940 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 18:03:12.141069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.141051 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 18:03:12.169264 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.169239 2582 apiserver.go:52] "Watching apiserver" Sep 17 18:03:12.186298 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.186281 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 18:03:12.186669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.186649 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk","openshift-multus/multus-p6jdx","openshift-network-diagnostics/network-check-target-wrnpc","openshift-network-operator/iptables-alerter-zpcvp","openshift-cluster-node-tuning-operator/tuned-j6l9f","openshift-dns/node-resolver-spg6t","openshift-image-registry/node-ca-ktpwl","openshift-multus/multus-additional-cni-plugins-nxm8m","openshift-multus/network-metrics-daemon-ctrs8","openshift-ovn-kubernetes/ovnkube-node-kqs7r","kube-system/global-pull-secret-syncer-tzwqq","kube-system/konnectivity-agent-t4gc6"] Sep 17 18:03:12.187560 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.187546 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 18:03:12.187934 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.187918 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" Sep 17 18:03:12.188477 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.188461 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.189834 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.189813 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.191147 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.191130 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:12.191235 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.191193 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:12.191900 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.191881 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-socket-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.192013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.191901 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 18:03:12.192013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.191920 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-registration-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.192013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.191952 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-device-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.192013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.191981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.192013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192006 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.192189 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192029 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-sys-fs\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.192189 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192056 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8v8\" (UniqueName: \"kubernetes.io/projected/ad8723c7-abed-445f-a37c-8b57c187b757-kube-api-access-8r8v8\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.192589 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192570 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.192831 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192809 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 18:03:12.192955 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192939 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 18:03:12.193000 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192956 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 18:03:12.193000 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192991 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k4c76\"" Sep 17 18:03:12.193069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.192938 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 18:03:12.193069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.193037 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnbk2\"" Sep 17 18:03:12.193258 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.193235 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 18:03:12.193314 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.193300 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 18:03:12.194094 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.194077 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.195179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.195150 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ccbh2\"" Sep 17 18:03:12.195314 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.195165 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 18:03:12.195314 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.195167 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 18:03:12.195522 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.195506 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.195826 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.195810 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:03:12.196016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.196000 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:03:12.196316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.196300 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 18:03:12.196407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.196392 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lk6pt\"" Sep 17 18:03:12.196949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.196934 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.198522 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.198507 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.200133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.200114 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:12.200206 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.200176 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:12.201175 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.201038 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 18:03:12.201175 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.201054 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 18:03:12.201175 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.201063 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 18:03:12.201175 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.201076 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 18:03:12.201382 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.201306 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wpp9z\"" Sep 17 18:03:12.201599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.201576 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 18:03:12.202154 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.201838 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r5f56\"" Sep 17 18:03:12.202154 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.202078 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 18:03:12.202584 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.202563 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 18:03:12.202782 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.202756 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2nzwr\"" Sep 17 18:03:12.203059 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.203040 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 18:03:12.203182 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.203168 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" Sep 17 18:03:12.203350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.203331 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.206731 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.206707 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.206876 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.206834 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:12.207267 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.207248 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:03:12.207952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.207843 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 18:03:12.207952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.207893 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 18:03:12.207952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.207831 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 18:03:12.208113 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.208085 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 18:03:12.208224 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.208207 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9stnd\"" Sep 17 18:03:12.208268 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.208232 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 18:03:12.208268 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.208217 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 18:03:12.208602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.208586 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 18:03:12.208946 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.208931 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal"] Sep 17 18:03:12.209024 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.209011 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:12.211204 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.211188 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 18:03:12.211321 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.211306 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 18:03:12.211408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.211391 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9tgtr\"" Sep 17 18:03:12.213929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.213912 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 18:03:12.214008 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.213912 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal"] Sep 17 18:03:12.217653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.217363 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:58:11 +0000 UTC" deadline="2028-04-10 01:39:56.979684115 +0000 UTC" Sep 17 18:03:12.217653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.217409 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13687h36m44.76227885s" Sep 17 18:03:12.237103 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.237083 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m29nt" Sep 17 18:03:12.246478 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.246457 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m29nt" Sep 17 18:03:12.288839 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.288818 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 18:03:12.292988 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.292969 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6tvb\" (UniqueName: \"kubernetes.io/projected/23d3772b-6060-4903-be8a-49a0704c9a3e-kube-api-access-p6tvb\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.293049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.292995 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-cni-multus\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.293049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.293049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293031 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z666k\" (UniqueName: \"kubernetes.io/projected/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-kube-api-access-z666k\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.293174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293055 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-systemd-units\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.293174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293077 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.293174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293092 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23c993ac-92ba-4856-a568-4eb62f489a61-host-slash\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.293174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293106 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-system-cni-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.293174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-tmp\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.293407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293189 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-system-cni-dir\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.293407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293235 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-kubelet\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.293407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293267 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-node-log\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.293407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovnkube-script-lib\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.293407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/912b1cfc-fb70-47aa-9558-f12d1d3f4416-kubelet-config\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.293407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293341 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:12.293407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293371 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-conf-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.293407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293395 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-lib-modules\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-var-lib-kubelet\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7398a155-5335-448b-b8b5-c86efaf5e6b6-serviceca\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhfp\" (UniqueName: \"kubernetes.io/projected/7398a155-5335-448b-b8b5-c86efaf5e6b6-kube-api-access-pfhfp\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293508 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293534 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-modprobe-d\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-systemd\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293583 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-run\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-ovn\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293615 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23c993ac-92ba-4856-a568-4eb62f489a61-iptables-alerter-script\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-cni-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293653 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-kubelet\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.293683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8v8\" (UniqueName: \"kubernetes.io/projected/ad8723c7-abed-445f-a37c-8b57c187b757-kube-api-access-8r8v8\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-socket-dir-parent\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-log-socket\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-netns\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-daemon-config\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293810 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-slash\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293842 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzgqx\" (UniqueName: \"kubernetes.io/projected/d9b01902-a855-4aec-8617-33e5e2ab66ea-kube-api-access-zzgqx\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293914 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntvbm\" (UniqueName: \"kubernetes.io/projected/23c993ac-92ba-4856-a568-4eb62f489a61-kube-api-access-ntvbm\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293937 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c293e70-3fd6-4e79-86c7-123b5f7c7dee-agent-certs\") pod \"konnectivity-agent-t4gc6\" (UID: \"8c293e70-3fd6-4e79-86c7-123b5f7c7dee\") " pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysconfig\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293965 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.293983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovnkube-config\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294012 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-k8s-cni-cncf-io\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294029 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-cni-bin\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294044 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-etc-kubernetes\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294065 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cnibin\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294083 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-etc-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294113 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-cni-bin\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-cni-netd\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294170 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23d3772b-6060-4903-be8a-49a0704c9a3e-hosts-file\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrwh\" (UniqueName: \"kubernetes.io/projected/dce4d64f-b353-4a67-bd2d-e92d79760b4e-kube-api-access-sfrwh\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294237 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-kubernetes\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294270 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovn-node-metrics-cert\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c293e70-3fd6-4e79-86c7-123b5f7c7dee-konnectivity-ca\") pod \"konnectivity-agent-t4gc6\" (UID: \"8c293e70-3fd6-4e79-86c7-123b5f7c7dee\") " pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294301 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-tuned\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-env-overrides\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-cnibin\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce4d64f-b353-4a67-bd2d-e92d79760b4e-cni-binary-copy\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.295007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294388 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-hostroot\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294401 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysctl-conf\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294415 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-sys\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7398a155-5335-448b-b8b5-c86efaf5e6b6-host\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294438 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294465 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-systemd\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294495 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-var-lib-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294527 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-sys-fs\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d3772b-6060-4903-be8a-49a0704c9a3e-tmp-dir\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gbvc\" (UniqueName: \"kubernetes.io/projected/9dd86bb7-9b68-4957-87ba-905e2ffe4948-kube-api-access-5gbvc\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294579 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-sys-fs\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294596 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-socket-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294679 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-registration-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-device-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-multus-certs\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-socket-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dslf\" (UniqueName: \"kubernetes.io/projected/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-kube-api-access-5dslf\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.295944 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294759 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-registration-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295944 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294773 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad8723c7-abed-445f-a37c-8b57c187b757-device-dir\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.295944 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294783 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.295944 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294809 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/912b1cfc-fb70-47aa-9558-f12d1d3f4416-dbus\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.295944 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysctl-d\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.295944 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-host\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.295944 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.294895 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-run-netns\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.302920 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.302896 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 18:03:12.306003 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.305975 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8v8\" (UniqueName: \"kubernetes.io/projected/ad8723c7-abed-445f-a37c-8b57c187b757-kube-api-access-8r8v8\") pod \"aws-ebs-csi-driver-node-ljnpk\" (UID: \"ad8723c7-abed-445f-a37c-8b57c187b757\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.356431 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.356400 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a81d5fc529fcc24ca4d74829ae35e9.slice/crio-299e32c553d160a406edef56845a8a7a02d5879214e7be9897d6f546bdc24438 WatchSource:0}: Error finding container 299e32c553d160a406edef56845a8a7a02d5879214e7be9897d6f546bdc24438: Status 404 returned error can't find the container with id 299e32c553d160a406edef56845a8a7a02d5879214e7be9897d6f546bdc24438 Sep 17 18:03:12.356972 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.356952 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70d28bf90c68089cd4d8c2ce3d2b7fc.slice/crio-3693a782b46dfb63708ea80b0bbd184232aa9c65a8267386140e310f9844d441 WatchSource:0}: Error finding container 3693a782b46dfb63708ea80b0bbd184232aa9c65a8267386140e310f9844d441: Status 404 returned error can't find the container with id 3693a782b46dfb63708ea80b0bbd184232aa9c65a8267386140e310f9844d441 Sep 17 18:03:12.362898 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.362883 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:03:12.384742 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.384723 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:03:12.395420 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-k8s-cni-cncf-io\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.395523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395426 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-cni-bin\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.395523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-etc-kubernetes\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.395523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395456 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cnibin\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.395523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-etc-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.395523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395523 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-etc-kubernetes\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395508 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-cni-bin\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-etc-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395532 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-cni-bin\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395564 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cnibin\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395570 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-k8s-cni-cncf-io\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395591 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-cni-netd\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395569 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-cni-bin\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395607 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395638 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-cni-netd\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23d3772b-6060-4903-be8a-49a0704c9a3e-hosts-file\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrwh\" (UniqueName: \"kubernetes.io/projected/dce4d64f-b353-4a67-bd2d-e92d79760b4e-kube-api-access-sfrwh\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-kubernetes\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395719 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/23d3772b-6060-4903-be8a-49a0704c9a3e-hosts-file\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.395728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395718 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395750 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovn-node-metrics-cert\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c293e70-3fd6-4e79-86c7-123b5f7c7dee-konnectivity-ca\") pod \"konnectivity-agent-t4gc6\" (UID: \"8c293e70-3fd6-4e79-86c7-123b5f7c7dee\") " pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-tuned\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-kubernetes\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395806 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395848 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-env-overrides\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-cnibin\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce4d64f-b353-4a67-bd2d-e92d79760b4e-cni-binary-copy\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395934 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-hostroot\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysctl-conf\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.395996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-sys\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396009 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-hostroot\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7398a155-5335-448b-b8b5-c86efaf5e6b6-host\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396066 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-cnibin\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396088 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-systemd\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.396398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396130 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-var-lib-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396147 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-sys\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d3772b-6060-4903-be8a-49a0704c9a3e-tmp-dir\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396189 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-systemd\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396205 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gbvc\" (UniqueName: \"kubernetes.io/projected/9dd86bb7-9b68-4957-87ba-905e2ffe4948-kube-api-access-5gbvc\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7398a155-5335-448b-b8b5-c86efaf5e6b6-host\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396234 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-multus-certs\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396351 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dslf\" (UniqueName: \"kubernetes.io/projected/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-kube-api-access-5dslf\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396387 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c293e70-3fd6-4e79-86c7-123b5f7c7dee-konnectivity-ca\") pod \"konnectivity-agent-t4gc6\" (UID: \"8c293e70-3fd6-4e79-86c7-123b5f7c7dee\") " pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/912b1cfc-fb70-47aa-9558-f12d1d3f4416-dbus\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396471 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-multus-certs\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysctl-d\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-var-lib-openvswitch\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-host\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396360 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-env-overrides\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-run-netns\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.397308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396550 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dce4d64f-b353-4a67-bd2d-e92d79760b4e-cni-binary-copy\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysctl-conf\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396654 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-host\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/912b1cfc-fb70-47aa-9558-f12d1d3f4416-dbus\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.396654 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396673 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysctl-d\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.396751 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret podName:912b1cfc-fb70-47aa-9558-f12d1d3f4416 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:12.89671675 +0000 UTC m=+3.960660706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret") pod "global-pull-secret-syncer-tzwqq" (UID: "912b1cfc-fb70-47aa-9558-f12d1d3f4416") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396773 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-run-netns\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6tvb\" (UniqueName: \"kubernetes.io/projected/23d3772b-6060-4903-be8a-49a0704c9a3e-kube-api-access-p6tvb\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-cni-multus\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396888 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z666k\" (UniqueName: \"kubernetes.io/projected/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-kube-api-access-z666k\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396942 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-cni-multus\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396944 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-systemd-units\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396997 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397011 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23d3772b-6060-4903-be8a-49a0704c9a3e-tmp-dir\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.398165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397017 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23c993ac-92ba-4856-a568-4eb62f489a61-host-slash\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397046 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/23c993ac-92ba-4856-a568-4eb62f489a61-host-slash\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.396991 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-systemd-units\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397044 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-run-ovn-kubernetes\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397079 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-system-cni-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-tmp\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-system-cni-dir\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397123 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-system-cni-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-kubelet\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397185 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-system-cni-dir\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-node-log\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovnkube-script-lib\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397244 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/912b1cfc-fb70-47aa-9558-f12d1d3f4416-kubelet-config\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-kubelet\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397269 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397221 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-node-log\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-conf-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.398805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-lib-modules\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397342 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/912b1cfc-fb70-47aa-9558-f12d1d3f4416-kubelet-config\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397363 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-var-lib-kubelet\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397385 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-conf-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397394 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7398a155-5335-448b-b8b5-c86efaf5e6b6-serviceca\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397416 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-cni-binary-copy\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhfp\" (UniqueName: \"kubernetes.io/projected/7398a155-5335-448b-b8b5-c86efaf5e6b6-kube-api-access-pfhfp\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397477 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-var-lib-kubelet\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397478 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397494 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-lib-modules\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397506 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-modprobe-d\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-systemd\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397585 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-run\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397615 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-ovn\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23c993ac-92ba-4856-a568-4eb62f489a61-iptables-alerter-script\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397674 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-cni-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.399363 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397689 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-modprobe-d\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397697 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-kubelet\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397732 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-socket-dir-parent\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-log-socket\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-run-ovn\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397785 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-netns\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-daemon-config\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-slash\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzgqx\" (UniqueName: \"kubernetes.io/projected/d9b01902-a855-4aec-8617-33e5e2ab66ea-kube-api-access-zzgqx\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397901 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7398a155-5335-448b-b8b5-c86efaf5e6b6-serviceca\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntvbm\" (UniqueName: \"kubernetes.io/projected/23c993ac-92ba-4856-a568-4eb62f489a61-kube-api-access-ntvbm\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c293e70-3fd6-4e79-86c7-123b5f7c7dee-agent-certs\") pod \"konnectivity-agent-t4gc6\" (UID: \"8c293e70-3fd6-4e79-86c7-123b5f7c7dee\") " pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysconfig\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397970 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-cni-dir\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397960 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovnkube-script-lib\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovnkube-config\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398011 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-run-netns\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.399957 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.398075 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.398134 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs podName:9dd86bb7-9b68-4957-87ba-905e2ffe4948 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:12.898115184 +0000 UTC m=+3.962059139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs") pod "network-metrics-daemon-ctrs8" (UID: "9dd86bb7-9b68-4957-87ba-905e2ffe4948") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398179 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-systemd\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398230 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-socket-dir-parent\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398244 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398268 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-log-socket\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.397759 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dce4d64f-b353-4a67-bd2d-e92d79760b4e-host-var-lib-kubelet\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398298 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/23c993ac-92ba-4856-a568-4eb62f489a61-iptables-alerter-script\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398304 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-run\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398330 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b01902-a855-4aec-8617-33e5e2ab66ea-host-slash\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398395 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-sysconfig\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovnkube-config\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398478 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-etc-tuned\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398700 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dce4d64f-b353-4a67-bd2d-e92d79760b4e-multus-daemon-config\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.398926 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9b01902-a855-4aec-8617-33e5e2ab66ea-ovn-node-metrics-cert\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.399913 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-tmp\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.400456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.400261 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c293e70-3fd6-4e79-86c7-123b5f7c7dee-agent-certs\") pod \"konnectivity-agent-t4gc6\" (UID: \"8c293e70-3fd6-4e79-86c7-123b5f7c7dee\") " pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:12.404833 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.404808 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dslf\" (UniqueName: \"kubernetes.io/projected/70a2f348-ae7a-4e63-8ed6-c06ec02a0c13-kube-api-access-5dslf\") pod \"tuned-j6l9f\" (UID: \"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13\") " pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.404943 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.404876 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrwh\" (UniqueName: \"kubernetes.io/projected/dce4d64f-b353-4a67-bd2d-e92d79760b4e-kube-api-access-sfrwh\") pod \"multus-p6jdx\" (UID: \"dce4d64f-b353-4a67-bd2d-e92d79760b4e\") " pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.405077 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.405060 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gbvc\" (UniqueName: \"kubernetes.io/projected/9dd86bb7-9b68-4957-87ba-905e2ffe4948-kube-api-access-5gbvc\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:12.406475 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.406460 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:12.406556 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.406477 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:12.406556 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.406490 2582 projected.go:196] Error preparing data for projected volume kube-api-access-cx72z for pod openshift-network-diagnostics/network-check-target-wrnpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:12.406556 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.406545 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z podName:e8f6ca4e-3cc1-4d22-820e-afc5cdf98276 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:12.906520581 +0000 UTC m=+3.970464517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cx72z" (UniqueName: "kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z") pod "network-check-target-wrnpc" (UID: "e8f6ca4e-3cc1-4d22-820e-afc5cdf98276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:12.408048 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.408026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzgqx\" (UniqueName: \"kubernetes.io/projected/d9b01902-a855-4aec-8617-33e5e2ab66ea-kube-api-access-zzgqx\") pod \"ovnkube-node-kqs7r\" (UID: \"d9b01902-a855-4aec-8617-33e5e2ab66ea\") " pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.408562 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.408542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z666k\" (UniqueName: \"kubernetes.io/projected/3bfa9ffe-01f9-476f-8c41-4d8e60aaba75-kube-api-access-z666k\") pod \"multus-additional-cni-plugins-nxm8m\" (UID: \"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75\") " pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.408712 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.408695 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhfp\" (UniqueName: \"kubernetes.io/projected/7398a155-5335-448b-b8b5-c86efaf5e6b6-kube-api-access-pfhfp\") pod \"node-ca-ktpwl\" (UID: \"7398a155-5335-448b-b8b5-c86efaf5e6b6\") " pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.408932 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.408917 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntvbm\" (UniqueName: \"kubernetes.io/projected/23c993ac-92ba-4856-a568-4eb62f489a61-kube-api-access-ntvbm\") pod \"iptables-alerter-zpcvp\" (UID: \"23c993ac-92ba-4856-a568-4eb62f489a61\") " pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.409015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.408998 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6tvb\" (UniqueName: \"kubernetes.io/projected/23d3772b-6060-4903-be8a-49a0704c9a3e-kube-api-access-p6tvb\") pod \"node-resolver-spg6t\" (UID: \"23d3772b-6060-4903-be8a-49a0704c9a3e\") " pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.521139 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.521092 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" Sep 17 18:03:12.528788 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.528766 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8723c7_abed_445f_a37c_8b57c187b757.slice/crio-f2e348e4380bed24e8153169ee172a9deb0e5e4c67e65543078e924a6980168e WatchSource:0}: Error finding container f2e348e4380bed24e8153169ee172a9deb0e5e4c67e65543078e924a6980168e: Status 404 returned error can't find the container with id f2e348e4380bed24e8153169ee172a9deb0e5e4c67e65543078e924a6980168e Sep 17 18:03:12.538123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.538107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p6jdx" Sep 17 18:03:12.542755 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.542736 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpcvp" Sep 17 18:03:12.544599 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.544578 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce4d64f_b353_4a67_bd2d_e92d79760b4e.slice/crio-05cf1bd7bd6f6096560e523a843b83cf97decbb027a47e9caea5a7356a934b22 WatchSource:0}: Error finding container 05cf1bd7bd6f6096560e523a843b83cf97decbb027a47e9caea5a7356a934b22: Status 404 returned error can't find the container with id 05cf1bd7bd6f6096560e523a843b83cf97decbb027a47e9caea5a7356a934b22 Sep 17 18:03:12.549995 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.549975 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c993ac_92ba_4856_a568_4eb62f489a61.slice/crio-5e918e282a73cb27fa30353ef7e001139addf868f3a608ef007070a53d8f22de WatchSource:0}: Error finding container 5e918e282a73cb27fa30353ef7e001139addf868f3a608ef007070a53d8f22de: Status 404 returned error can't find the container with id 5e918e282a73cb27fa30353ef7e001139addf868f3a608ef007070a53d8f22de Sep 17 18:03:12.554326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.554305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" Sep 17 18:03:12.561936 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.561916 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a2f348_ae7a_4e63_8ed6_c06ec02a0c13.slice/crio-1214233a004eab2e857075f3e63f35bbd28398fabcce8bee47be4434aff989ec WatchSource:0}: Error finding container 1214233a004eab2e857075f3e63f35bbd28398fabcce8bee47be4434aff989ec: Status 404 returned error can't find the container with id 1214233a004eab2e857075f3e63f35bbd28398fabcce8bee47be4434aff989ec Sep 17 18:03:12.587706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.587683 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-spg6t" Sep 17 18:03:12.594501 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.594479 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d3772b_6060_4903_be8a_49a0704c9a3e.slice/crio-6fd2e5fc3615f057b9b9d9acddb489cb1621c4f40036de137323df8bca91d763 WatchSource:0}: Error finding container 6fd2e5fc3615f057b9b9d9acddb489cb1621c4f40036de137323df8bca91d763: Status 404 returned error can't find the container with id 6fd2e5fc3615f057b9b9d9acddb489cb1621c4f40036de137323df8bca91d763 Sep 17 18:03:12.605361 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.605346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ktpwl" Sep 17 18:03:12.611799 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.611778 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7398a155_5335_448b_b8b5_c86efaf5e6b6.slice/crio-c0a31a74bff33c20cfb9e0882157535f13b53e7bd8482769b19c8be65f84cc5b WatchSource:0}: Error finding container c0a31a74bff33c20cfb9e0882157535f13b53e7bd8482769b19c8be65f84cc5b: Status 404 returned error can't find the container with id c0a31a74bff33c20cfb9e0882157535f13b53e7bd8482769b19c8be65f84cc5b Sep 17 18:03:12.633016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.632997 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" Sep 17 18:03:12.639786 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.639768 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bfa9ffe_01f9_476f_8c41_4d8e60aaba75.slice/crio-bccaddb848bc5029a4268c0c549a18028460d84523542c8dc611a2941713d931 WatchSource:0}: Error finding container bccaddb848bc5029a4268c0c549a18028460d84523542c8dc611a2941713d931: Status 404 returned error can't find the container with id bccaddb848bc5029a4268c0c549a18028460d84523542c8dc611a2941713d931 Sep 17 18:03:12.652219 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.652200 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:12.656056 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.656032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:12.658925 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.658897 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b01902_a855_4aec_8617_33e5e2ab66ea.slice/crio-d46a61613b9ebc652512f35b344bb21526a6eb9a8388dc554d352be90ee9f9ab WatchSource:0}: Error finding container d46a61613b9ebc652512f35b344bb21526a6eb9a8388dc554d352be90ee9f9ab: Status 404 returned error can't find the container with id d46a61613b9ebc652512f35b344bb21526a6eb9a8388dc554d352be90ee9f9ab Sep 17 18:03:12.663013 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:12.662996 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c293e70_3fd6_4e79_86c7_123b5f7c7dee.slice/crio-ae80c8f8e48aed57c01f69da00b1e0740a7013f771f1d90e941f2834c9eaee88 WatchSource:0}: Error finding container ae80c8f8e48aed57c01f69da00b1e0740a7013f771f1d90e941f2834c9eaee88: Status 404 returned error can't find the container with id ae80c8f8e48aed57c01f69da00b1e0740a7013f771f1d90e941f2834c9eaee88 Sep 17 18:03:12.903296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.903246 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:12.903448 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:12.903337 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:12.903448 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.903389 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:12.903448 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.903439 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:12.903580 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.903453 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs podName:9dd86bb7-9b68-4957-87ba-905e2ffe4948 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:13.903438476 +0000 UTC m=+4.967382412 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs") pod "network-metrics-daemon-ctrs8" (UID: "9dd86bb7-9b68-4957-87ba-905e2ffe4948") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:12.903580 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:12.903487 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret podName:912b1cfc-fb70-47aa-9558-f12d1d3f4416 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:13.903472933 +0000 UTC m=+4.967416887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret") pod "global-pull-secret-syncer-tzwqq" (UID: "912b1cfc-fb70-47aa-9558-f12d1d3f4416") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:13.004644 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.004613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:13.004828 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.004806 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:13.004926 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.004839 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:13.004926 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.004853 2582 projected.go:196] Error preparing data for projected volume kube-api-access-cx72z for pod openshift-network-diagnostics/network-check-target-wrnpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:13.005032 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.004930 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z podName:e8f6ca4e-3cc1-4d22-820e-afc5cdf98276 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:14.004911019 +0000 UTC m=+5.068854974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cx72z" (UniqueName: "kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z") pod "network-check-target-wrnpc" (UID: "e8f6ca4e-3cc1-4d22-820e-afc5cdf98276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:13.248460 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.248348 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:58:12 +0000 UTC" deadline="2028-04-30 04:31:30.267531992 +0000 UTC" Sep 17 18:03:13.248460 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.248413 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14170h28m17.019122517s" Sep 17 18:03:13.275178 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.275148 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 18:03:13.342538 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.341384 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:13.342538 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.341516 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:13.353739 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.353681 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ktpwl" event={"ID":"7398a155-5335-448b-b8b5-c86efaf5e6b6","Type":"ContainerStarted","Data":"c0a31a74bff33c20cfb9e0882157535f13b53e7bd8482769b19c8be65f84cc5b"} Sep 17 18:03:13.357432 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.357399 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" event={"ID":"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13","Type":"ContainerStarted","Data":"1214233a004eab2e857075f3e63f35bbd28398fabcce8bee47be4434aff989ec"} Sep 17 18:03:13.366470 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.366384 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpcvp" event={"ID":"23c993ac-92ba-4856-a568-4eb62f489a61","Type":"ContainerStarted","Data":"5e918e282a73cb27fa30353ef7e001139addf868f3a608ef007070a53d8f22de"} Sep 17 18:03:13.372388 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.372364 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p6jdx" event={"ID":"dce4d64f-b353-4a67-bd2d-e92d79760b4e","Type":"ContainerStarted","Data":"05cf1bd7bd6f6096560e523a843b83cf97decbb027a47e9caea5a7356a934b22"} Sep 17 18:03:13.374823 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.374732 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" event={"ID":"73a81d5fc529fcc24ca4d74829ae35e9","Type":"ContainerStarted","Data":"299e32c553d160a406edef56845a8a7a02d5879214e7be9897d6f546bdc24438"} Sep 17 18:03:13.383553 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.383523 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" event={"ID":"a70d28bf90c68089cd4d8c2ce3d2b7fc","Type":"ContainerStarted","Data":"3693a782b46dfb63708ea80b0bbd184232aa9c65a8267386140e310f9844d441"} Sep 17 18:03:13.387803 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.387779 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t4gc6" event={"ID":"8c293e70-3fd6-4e79-86c7-123b5f7c7dee","Type":"ContainerStarted","Data":"ae80c8f8e48aed57c01f69da00b1e0740a7013f771f1d90e941f2834c9eaee88"} Sep 17 18:03:13.394119 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.394050 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"d46a61613b9ebc652512f35b344bb21526a6eb9a8388dc554d352be90ee9f9ab"} Sep 17 18:03:13.400290 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.400265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerStarted","Data":"bccaddb848bc5029a4268c0c549a18028460d84523542c8dc611a2941713d931"} Sep 17 18:03:13.418506 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.418478 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-spg6t" event={"ID":"23d3772b-6060-4903-be8a-49a0704c9a3e","Type":"ContainerStarted","Data":"6fd2e5fc3615f057b9b9d9acddb489cb1621c4f40036de137323df8bca91d763"} Sep 17 18:03:13.424955 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.424912 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" event={"ID":"ad8723c7-abed-445f-a37c-8b57c187b757","Type":"ContainerStarted","Data":"f2e348e4380bed24e8153169ee172a9deb0e5e4c67e65543078e924a6980168e"} Sep 17 18:03:13.911113 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.911076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:13.911288 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:13.911150 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:13.911288 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.911283 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:13.911429 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.911349 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret podName:912b1cfc-fb70-47aa-9558-f12d1d3f4416 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:15.911327408 +0000 UTC m=+6.975271363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret") pod "global-pull-secret-syncer-tzwqq" (UID: "912b1cfc-fb70-47aa-9558-f12d1d3f4416") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:13.911761 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.911735 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:13.911833 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:13.911788 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs podName:9dd86bb7-9b68-4957-87ba-905e2ffe4948 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:15.911772839 +0000 UTC m=+6.975716779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs") pod "network-metrics-daemon-ctrs8" (UID: "9dd86bb7-9b68-4957-87ba-905e2ffe4948") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:14.011692 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:14.011648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:14.011951 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:14.011828 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:14.011951 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:14.011848 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:14.011951 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:14.011876 2582 projected.go:196] Error preparing data for projected volume kube-api-access-cx72z for pod openshift-network-diagnostics/network-check-target-wrnpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:14.011951 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:14.011939 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z podName:e8f6ca4e-3cc1-4d22-820e-afc5cdf98276 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:16.01191967 +0000 UTC m=+7.075863611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cx72z" (UniqueName: "kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z") pod "network-check-target-wrnpc" (UID: "e8f6ca4e-3cc1-4d22-820e-afc5cdf98276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:14.249107 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:14.249016 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:58:12 +0000 UTC" deadline="2028-06-08 05:11:06.221470756 +0000 UTC" Sep 17 18:03:14.249107 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:14.249057 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15107h7m51.972418536s" Sep 17 18:03:14.339543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:14.339511 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:14.339683 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:14.339641 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:14.340076 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:14.340058 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:14.340185 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:14.340165 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:15.340560 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:15.340075 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:15.340560 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:15.340196 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:15.928039 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:15.927994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:15.928228 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:15.928081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:15.928286 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:15.928236 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:15.928335 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:15.928302 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs podName:9dd86bb7-9b68-4957-87ba-905e2ffe4948 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:19.928282363 +0000 UTC m=+10.992226324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs") pod "network-metrics-daemon-ctrs8" (UID: "9dd86bb7-9b68-4957-87ba-905e2ffe4948") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:15.928819 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:15.928697 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:15.928819 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:15.928754 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret podName:912b1cfc-fb70-47aa-9558-f12d1d3f4416 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:19.928737744 +0000 UTC m=+10.992681699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret") pod "global-pull-secret-syncer-tzwqq" (UID: "912b1cfc-fb70-47aa-9558-f12d1d3f4416") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:16.030140 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:16.029466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:16.030140 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:16.029688 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:16.030140 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:16.029708 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:16.030140 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:16.029721 2582 projected.go:196] Error preparing data for projected volume kube-api-access-cx72z for pod openshift-network-diagnostics/network-check-target-wrnpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:16.030140 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:16.029782 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z podName:e8f6ca4e-3cc1-4d22-820e-afc5cdf98276 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:20.029764785 +0000 UTC m=+11.093708736 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cx72z" (UniqueName: "kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z") pod "network-check-target-wrnpc" (UID: "e8f6ca4e-3cc1-4d22-820e-afc5cdf98276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:16.339141 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:16.339047 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:16.339312 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:16.339194 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:16.339911 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:16.339633 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:16.339911 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:16.339848 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:17.338952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:17.338918 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:17.339415 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:17.339058 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:18.338991 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:18.338958 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:18.339428 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:18.338998 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:18.339428 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:18.339113 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:18.339428 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:18.339219 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:19.339214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:19.339183 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:19.339651 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:19.339313 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:19.956506 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:19.956460 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:19.956689 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:19.956548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:19.956752 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:19.956685 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:19.956752 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:19.956748 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs podName:9dd86bb7-9b68-4957-87ba-905e2ffe4948 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:27.956728744 +0000 UTC m=+19.020672697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs") pod "network-metrics-daemon-ctrs8" (UID: "9dd86bb7-9b68-4957-87ba-905e2ffe4948") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:19.956971 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:19.956937 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:19.957036 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:19.957007 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret podName:912b1cfc-fb70-47aa-9558-f12d1d3f4416 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:27.95699335 +0000 UTC m=+19.020937289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret") pod "global-pull-secret-syncer-tzwqq" (UID: "912b1cfc-fb70-47aa-9558-f12d1d3f4416") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:20.057096 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:20.057006 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:20.057261 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:20.057233 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:20.057261 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:20.057252 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:20.057375 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:20.057265 2582 projected.go:196] Error preparing data for projected volume kube-api-access-cx72z for pod openshift-network-diagnostics/network-check-target-wrnpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:20.057375 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:20.057325 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z podName:e8f6ca4e-3cc1-4d22-820e-afc5cdf98276 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:28.057305133 +0000 UTC m=+19.121249085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cx72z" (UniqueName: "kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z") pod "network-check-target-wrnpc" (UID: "e8f6ca4e-3cc1-4d22-820e-afc5cdf98276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:20.339645 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:20.339571 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:20.340047 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:20.339569 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:20.340047 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:20.339731 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:20.340047 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:20.339776 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:21.339616 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:21.339586 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:21.339813 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:21.339690 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:22.338938 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:22.338840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:22.339105 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:22.338987 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:22.339162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:22.339143 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:22.339322 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:22.339269 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:23.343430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:23.343401 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:23.343910 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:23.343499 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:24.339029 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:24.338990 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:24.339298 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:24.339032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:24.339298 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:24.339120 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:24.339298 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:24.339236 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:25.339554 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:25.339518 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:25.340052 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:25.339640 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:26.339487 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:26.339460 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:26.339487 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:26.339476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:26.339892 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:26.339572 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:26.339892 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:26.339693 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:27.343471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.343137 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:27.344025 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:27.343557 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:27.452014 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.451930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" event={"ID":"a70d28bf90c68089cd4d8c2ce3d2b7fc","Type":"ContainerStarted","Data":"1d64c068d681a64430cfefff67e6f92dc58e978c7ae6f85eb098e5d64b093f78"} Sep 17 18:03:27.453803 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.453769 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t4gc6" event={"ID":"8c293e70-3fd6-4e79-86c7-123b5f7c7dee","Type":"ContainerStarted","Data":"35ab2c8d87a07dc39e8ce16554fc8c8220b1167497a70e819dac3e337c9ae532"} Sep 17 18:03:27.457287 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.457267 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:03:27.457764 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.457741 2582 generic.go:356] "Generic (PLEG): container finished" podID="d9b01902-a855-4aec-8617-33e5e2ab66ea" containerID="eb57543c3dbe51c5e3c806c2bc4e8e0f76615338d6ef60d6ced246feb52f3bb9" exitCode=1 Sep 17 18:03:27.457857 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.457810 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"f5773320525bbb76cd8d0032de54d6eedbdf30d6d81eae4fa2f97b66620439b7"} Sep 17 18:03:27.457947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.457934 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"f2dc4e0c1ef5b8f8d347506fc0b5c3ba7fd9b11bd8b72d6c707d14d233d27501"} Sep 17 18:03:27.458022 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.458001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"1d83a24c66d6f7f6c0d15fd1dc58ffcbdd4700f27f138e09d26ef67476cefaa3"} Sep 17 18:03:27.458137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.458055 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"f54345a61ea0ef78ef54ce9b34c56dd12e092f9fb897a825aeb3c89e5813e51b"} Sep 17 18:03:27.458137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.458119 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerDied","Data":"eb57543c3dbe51c5e3c806c2bc4e8e0f76615338d6ef60d6ced246feb52f3bb9"} Sep 17 18:03:27.458279 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.458179 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"05bdefa705ad5dfc523a1849655f256e0afadb36fca12e54b694fb0bfe14f2b2"} Sep 17 18:03:27.459663 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.459639 2582 generic.go:356] "Generic (PLEG): container finished" podID="3bfa9ffe-01f9-476f-8c41-4d8e60aaba75" containerID="bc6f8b0c40f24cb8d8986fafc93945be3765d8cd711597f0d3ee4f54bc21fa17" exitCode=0 Sep 17 18:03:27.459765 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.459729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerDied","Data":"bc6f8b0c40f24cb8d8986fafc93945be3765d8cd711597f0d3ee4f54bc21fa17"} Sep 17 18:03:27.461910 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.461850 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-spg6t" event={"ID":"23d3772b-6060-4903-be8a-49a0704c9a3e","Type":"ContainerStarted","Data":"75ef773d3c140a206b0acfa1909b06f7006b0cf25217a622047298485224a2db"} Sep 17 18:03:27.463704 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.463678 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" event={"ID":"ad8723c7-abed-445f-a37c-8b57c187b757","Type":"ContainerStarted","Data":"e3e4c0e0cb5764263b4f301280203b6ff5dec55dc38e39a3a4547d475a03432f"} Sep 17 18:03:27.465444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.465422 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ktpwl" event={"ID":"7398a155-5335-448b-b8b5-c86efaf5e6b6","Type":"ContainerStarted","Data":"4a89b95daa455a51be872e505007a330ba0ab8f7929c2f2edc2df428f83ca315"} Sep 17 18:03:27.467217 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.467197 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" event={"ID":"70a2f348-ae7a-4e63-8ed6-c06ec02a0c13","Type":"ContainerStarted","Data":"83ddea43a30d41bab75bc7b572dd470be6cd9bf9c0a491f4ef0894c16d6dfea3"} Sep 17 18:03:27.469141 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.469118 2582 generic.go:356] "Generic (PLEG): container finished" podID="73a81d5fc529fcc24ca4d74829ae35e9" containerID="d19b516334b9147a9561f80b69f74443cf1750d248326b081c3a7d9e2f50fe83" exitCode=0 Sep 17 18:03:27.469265 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.469154 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" event={"ID":"73a81d5fc529fcc24ca4d74829ae35e9","Type":"ContainerDied","Data":"d19b516334b9147a9561f80b69f74443cf1750d248326b081c3a7d9e2f50fe83"} Sep 17 18:03:27.472942 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.472514 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-104.ec2.internal" podStartSLOduration=15.472499327 podStartE2EDuration="15.472499327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:27.47202032 +0000 UTC m=+18.535964274" watchObservedRunningTime="2026-09-17 18:03:27.472499327 +0000 UTC m=+18.536443284" Sep 17 18:03:27.493982 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.493938 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ktpwl" podStartSLOduration=2.752677672 podStartE2EDuration="16.493920852s" totalImagesPullingTime="13.74124318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:27.493739938 +0000 UTC m=+18.557683896" watchObservedRunningTime="2026-09-17 18:03:27.493920852 +0000 UTC m=+18.557864815" Sep 17 18:03:27.544304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.544264 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j6l9f" podStartSLOduration=2.739944053 podStartE2EDuration="16.544250547s" totalImagesPullingTime="13.804306494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:27.544194493 +0000 UTC m=+18.608138452" watchObservedRunningTime="2026-09-17 18:03:27.544250547 +0000 UTC m=+18.608194505" Sep 17 18:03:27.562201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.562154 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-spg6t" podStartSLOduration=2.8053791930000003 podStartE2EDuration="16.562140616s" totalImagesPullingTime="13.756761423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:27.562092855 +0000 UTC m=+18.626036814" watchObservedRunningTime="2026-09-17 18:03:27.562140616 +0000 UTC m=+18.626084573" Sep 17 18:03:27.604113 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:27.603999 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t4gc6" podStartSLOduration=2.915653674 podStartE2EDuration="16.603979408s" totalImagesPullingTime="13.688325734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:27.578573724 +0000 UTC m=+18.642517698" watchObservedRunningTime="2026-09-17 18:03:27.603979408 +0000 UTC m=+18.667923366" Sep 17 18:03:28.021453 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.021362 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:28.021621 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.021491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:28.021621 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.021606 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:28.021734 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.021679 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret podName:912b1cfc-fb70-47aa-9558-f12d1d3f4416 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:44.021661448 +0000 UTC m=+35.085605390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret") pod "global-pull-secret-syncer-tzwqq" (UID: "912b1cfc-fb70-47aa-9558-f12d1d3f4416") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:28.021799 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.021741 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:28.021799 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.021774 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs podName:9dd86bb7-9b68-4957-87ba-905e2ffe4948 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:44.021761445 +0000 UTC m=+35.085705381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs") pod "network-metrics-daemon-ctrs8" (UID: "9dd86bb7-9b68-4957-87ba-905e2ffe4948") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:28.122084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.122052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:28.122273 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.122204 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:28.122273 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.122225 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:28.122273 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.122237 2582 projected.go:196] Error preparing data for projected volume kube-api-access-cx72z for pod openshift-network-diagnostics/network-check-target-wrnpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:28.122401 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.122301 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z podName:e8f6ca4e-3cc1-4d22-820e-afc5cdf98276 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:44.122282874 +0000 UTC m=+35.186226830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cx72z" (UniqueName: "kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z") pod "network-check-target-wrnpc" (UID: "e8f6ca4e-3cc1-4d22-820e-afc5cdf98276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:28.252650 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.252629 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 18:03:28.283151 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.282957 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:03:28.252645301Z","UUID":"ce4fd5cd-3e31-4436-bf80-0e807a86d62a","Handler":null,"Name":"","Endpoint":""} Sep 17 18:03:28.284573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.284529 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 18:03:28.284573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.284563 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 18:03:28.339009 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.338986 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:28.339009 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.339000 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:28.339211 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.339090 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:28.339267 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:28.339220 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:28.473556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.473523 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpcvp" event={"ID":"23c993ac-92ba-4856-a568-4eb62f489a61","Type":"ContainerStarted","Data":"4fcb0c689bc9917e907c7bb673b71eebff58c3b48332bd8c6ca5cb8b0e2fa237"} Sep 17 18:03:28.476414 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.476388 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" event={"ID":"73a81d5fc529fcc24ca4d74829ae35e9","Type":"ContainerStarted","Data":"4c8acf9da2f336007b7d9ac9ccc33630db9351fd275eae81336f2a044a49a777"} Sep 17 18:03:28.480089 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.480061 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" event={"ID":"ad8723c7-abed-445f-a37c-8b57c187b757","Type":"ContainerStarted","Data":"1c65a6058fcca2dfbf058566a1e8281ff8e77ae5f5715cd5a92e1b35ec491a7a"} Sep 17 18:03:28.498446 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.498404 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zpcvp" podStartSLOduration=3.6972148479999998 podStartE2EDuration="17.498388202s" totalImagesPullingTime="13.801173354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:28.498389114 +0000 UTC m=+19.562333073" watchObservedRunningTime="2026-09-17 18:03:28.498388202 +0000 UTC m=+19.562332159" Sep 17 18:03:28.933635 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.933602 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:28.934459 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.934437 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:28.964013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:28.963959 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-104.ec2.internal" podStartSLOduration=16.963935013 podStartE2EDuration="16.963935013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:28.525328256 +0000 UTC m=+19.589272216" watchObservedRunningTime="2026-09-17 18:03:28.963935013 +0000 UTC m=+20.027878976" Sep 17 18:03:29.339109 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:29.339071 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:29.339290 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:29.339207 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:29.483116 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:29.483085 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:29.483543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:29.483237 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t4gc6" Sep 17 18:03:30.339150 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:30.338938 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:30.339331 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:30.338971 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:30.339331 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:30.339252 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:30.339451 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:30.339344 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:31.339753 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:31.339723 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:31.340212 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:31.339813 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:32.339643 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:32.339609 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:32.339837 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:32.339619 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:32.339837 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:32.339725 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:32.340221 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:32.339855 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:33.339582 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:33.339549 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:33.339765 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:33.339685 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:34.338882 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:34.338830 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:34.339387 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:34.338991 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:34.339387 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:34.339038 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:34.339387 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:34.339148 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:35.339619 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:35.339587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:35.340049 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:35.339727 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:35.502202 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:35.502178 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:03:35.502533 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:35.502515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"0975341639f30811af97eb323fabc5a1d745ee7e18bce846c05f6586c9c0472c"} Sep 17 18:03:35.504477 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:35.504453 2582 generic.go:356] "Generic (PLEG): container finished" podID="3bfa9ffe-01f9-476f-8c41-4d8e60aaba75" containerID="47764891bad84b8107e762fcf38289f923667f509d6b09ec388954cc98818738" exitCode=0 Sep 17 18:03:35.504582 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:35.504523 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerDied","Data":"47764891bad84b8107e762fcf38289f923667f509d6b09ec388954cc98818738"} Sep 17 18:03:35.506688 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:35.506668 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" event={"ID":"ad8723c7-abed-445f-a37c-8b57c187b757","Type":"ContainerStarted","Data":"b5c5dd102d5ce5d146c439d68c19665747c9ba9fdb6a309b491636ce8ef9a6d6"} Sep 17 18:03:35.508045 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:35.508025 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p6jdx" event={"ID":"dce4d64f-b353-4a67-bd2d-e92d79760b4e","Type":"ContainerStarted","Data":"15e5c26c1fe33bd70182604a8e4bdaf2a71b9b9b1ff11c79aa2dd488ad9542f8"} Sep 17 18:03:35.584137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:35.583980 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljnpk" podStartSLOduration=3.209193342 podStartE2EDuration="24.583966629s" totalImagesPullingTime="21.374773287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:35.559171724 +0000 UTC m=+26.623115682" watchObservedRunningTime="2026-09-17 18:03:35.583966629 +0000 UTC m=+26.647910586" Sep 17 18:03:36.338798 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:36.338765 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:36.338947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:36.338888 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:36.338947 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:36.338902 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:36.339027 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:36.338973 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:36.512699 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:36.512664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerStarted","Data":"d67ad669359243cc23479ddb5c991c5c3d0a957f33e0ac0cc729f1c541e34263"} Sep 17 18:03:36.541419 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:36.541366 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p6jdx" podStartSLOduration=3.278344464 podStartE2EDuration="25.54134716s" totalImagesPullingTime="22.263002696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:35.583531781 +0000 UTC m=+26.647475739" watchObservedRunningTime="2026-09-17 18:03:36.54134716 +0000 UTC m=+27.605291119" Sep 17 18:03:37.339011 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:37.338933 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:37.339157 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:37.339040 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:37.520310 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:37.520281 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:03:37.521085 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:37.520648 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"cbf922a0a8536bf9bdbd4c5b264d6111404cdbea150cad3b4fe9f34a1de5219d"} Sep 17 18:03:37.521085 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:37.520908 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:37.521191 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:37.521121 2582 scope.go:117] "RemoveContainer" containerID="eb57543c3dbe51c5e3c806c2bc4e8e0f76615338d6ef60d6ced246feb52f3bb9" Sep 17 18:03:37.523072 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:37.523051 2582 generic.go:356] "Generic (PLEG): container finished" podID="3bfa9ffe-01f9-476f-8c41-4d8e60aaba75" containerID="d67ad669359243cc23479ddb5c991c5c3d0a957f33e0ac0cc729f1c541e34263" exitCode=0 Sep 17 18:03:37.523173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:37.523097 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerDied","Data":"d67ad669359243cc23479ddb5c991c5c3d0a957f33e0ac0cc729f1c541e34263"} Sep 17 18:03:37.536974 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:37.536955 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:38.339295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.339265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:38.339443 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:38.339357 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:38.339443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.339411 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:38.339526 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:38.339507 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:38.529331 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.529306 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:03:38.529749 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.529677 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" event={"ID":"d9b01902-a855-4aec-8617-33e5e2ab66ea","Type":"ContainerStarted","Data":"810adb364e56c03a6ec5fa8b6aebb748ade13b71d7f4f6568d395dc5b9663ea1"} Sep 17 18:03:38.529882 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.529843 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:38.530025 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.529933 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:38.532349 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.532321 2582 generic.go:356] "Generic (PLEG): container finished" podID="3bfa9ffe-01f9-476f-8c41-4d8e60aaba75" containerID="65870538ceb0594e4f184cbd0b8f314b2bde867644b7e5b7dd4b8ed5c372bfaa" exitCode=0 Sep 17 18:03:38.532459 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.532373 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerDied","Data":"65870538ceb0594e4f184cbd0b8f314b2bde867644b7e5b7dd4b8ed5c372bfaa"} Sep 17 18:03:38.544734 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.544717 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:03:38.574370 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:38.574329 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" podStartSLOduration=13.546080971 podStartE2EDuration="27.574316465s" totalImagesPullingTime="14.028235494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:38.573083821 +0000 UTC m=+29.637027778" watchObservedRunningTime="2026-09-17 18:03:38.574316465 +0000 UTC m=+29.638260420" Sep 17 18:03:39.339003 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:39.338971 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:39.339164 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:39.339110 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:40.339110 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:40.339078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:40.339543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:40.339078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:40.339543 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:40.339199 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:40.339543 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:40.339268 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:41.342515 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:41.342484 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:41.343011 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:41.342601 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:42.339697 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:42.339664 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:42.339852 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:42.339777 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:42.339852 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:42.339807 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:42.339955 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:42.339887 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:42.953099 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:42.952382 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tzwqq"] Sep 17 18:03:42.953099 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:42.952731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:42.953099 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:42.952845 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:42.956089 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:42.955834 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wrnpc"] Sep 17 18:03:42.956089 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:42.955951 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:42.956089 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:42.956052 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:42.974913 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:42.974829 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ctrs8"] Sep 17 18:03:42.975040 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:42.974950 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:42.975101 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:42.975044 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:44.051011 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:44.050979 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:44.051510 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:44.051047 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:44.051510 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.051145 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:44.051510 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.051228 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs podName:9dd86bb7-9b68-4957-87ba-905e2ffe4948 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.051207087 +0000 UTC m=+67.115151039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs") pod "network-metrics-daemon-ctrs8" (UID: "9dd86bb7-9b68-4957-87ba-905e2ffe4948") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 18:03:44.051510 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.051156 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:44.051510 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.051280 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret podName:912b1cfc-fb70-47aa-9558-f12d1d3f4416 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.051269772 +0000 UTC m=+67.115213722 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret") pod "global-pull-secret-syncer-tzwqq" (UID: "912b1cfc-fb70-47aa-9558-f12d1d3f4416") : object "kube-system"/"original-pull-secret" not registered Sep 17 18:03:44.152092 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:44.152059 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:44.152286 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.152262 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 18:03:44.152334 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.152298 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 18:03:44.152334 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.152313 2582 projected.go:196] Error preparing data for projected volume kube-api-access-cx72z for pod openshift-network-diagnostics/network-check-target-wrnpc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:44.152419 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.152385 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z podName:e8f6ca4e-3cc1-4d22-820e-afc5cdf98276 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.152365089 +0000 UTC m=+67.216309031 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cx72z" (UniqueName: "kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z") pod "network-check-target-wrnpc" (UID: "e8f6ca4e-3cc1-4d22-820e-afc5cdf98276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 18:03:44.339234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:44.339204 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:44.339397 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:44.339313 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:45.338730 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:45.338701 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:45.339149 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:45.338824 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:45.339149 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:45.338889 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:45.339149 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:45.338981 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:45.550083 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:45.550020 2582 generic.go:356] "Generic (PLEG): container finished" podID="3bfa9ffe-01f9-476f-8c41-4d8e60aaba75" containerID="afb2bb9e42916ea75bed6e49430b8db052fba2ee2f67b5a8d3ab93ede064d25e" exitCode=0 Sep 17 18:03:45.550175 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:45.550088 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerDied","Data":"afb2bb9e42916ea75bed6e49430b8db052fba2ee2f67b5a8d3ab93ede064d25e"} Sep 17 18:03:46.339063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:46.339032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:46.339422 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:46.339121 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tzwqq" podUID="912b1cfc-fb70-47aa-9558-f12d1d3f4416" Sep 17 18:03:46.554970 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:46.554937 2582 generic.go:356] "Generic (PLEG): container finished" podID="3bfa9ffe-01f9-476f-8c41-4d8e60aaba75" containerID="ca5fe02f139773d7544e4b29621f73c56132ca9aaf6b7ca32f81d2e7f9d4884e" exitCode=0 Sep 17 18:03:46.555123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:46.554983 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerDied","Data":"ca5fe02f139773d7544e4b29621f73c56132ca9aaf6b7ca32f81d2e7f9d4884e"} Sep 17 18:03:47.338983 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.338738 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:47.339089 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.338784 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:47.339337 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:47.339109 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ctrs8" podUID="9dd86bb7-9b68-4957-87ba-905e2ffe4948" Sep 17 18:03:47.339337 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:47.339143 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrnpc" podUID="e8f6ca4e-3cc1-4d22-820e-afc5cdf98276" Sep 17 18:03:47.560733 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.560660 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" event={"ID":"3bfa9ffe-01f9-476f-8c41-4d8e60aaba75","Type":"ContainerStarted","Data":"188dbfb8b0321e54fb2aa61ac2533c92368722294611c10320e32df2e51f0c8d"} Sep 17 18:03:47.590649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.590598 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nxm8m" podStartSLOduration=7.282015519 podStartE2EDuration="36.59057868s" totalImagesPullingTime="29.308563161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:47.589249674 +0000 UTC m=+38.653193635" watchObservedRunningTime="2026-09-17 18:03:47.59057868 +0000 UTC m=+38.654522639" Sep 17 18:03:47.748697 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.748673 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-104.ec2.internal" event="NodeReady" Sep 17 18:03:47.748893 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.748795 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 18:03:47.817620 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.817559 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-42kg8"] Sep 17 18:03:47.838802 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.838781 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4tllr"] Sep 17 18:03:47.838951 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.838932 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:47.851960 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.851893 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 18:03:47.851960 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.851940 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 18:03:47.852194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.852098 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dfk9\"" Sep 17 18:03:47.861208 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.861189 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-42kg8"] Sep 17 18:03:47.861297 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.861225 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4tllr"] Sep 17 18:03:47.861338 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.861301 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4tllr" Sep 17 18:03:47.864903 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.864885 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 18:03:47.865005 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.864943 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fht8w\"" Sep 17 18:03:47.865118 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.865103 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 18:03:47.865331 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.865316 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 18:03:47.983445 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.983418 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:47.983445 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.983448 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7gqj\" (UniqueName: \"kubernetes.io/projected/aeee9793-7091-4047-b99c-871e17adafa0-kube-api-access-q7gqj\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:47.983623 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.983480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tk5j\" (UniqueName: \"kubernetes.io/projected/bfa0606a-1289-4a5d-8271-f17430d5e740-kube-api-access-8tk5j\") pod \"ingress-canary-4tllr\" (UID: \"bfa0606a-1289-4a5d-8271-f17430d5e740\") " pod="openshift-ingress-canary/ingress-canary-4tllr" Sep 17 18:03:47.983623 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.983497 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa0606a-1289-4a5d-8271-f17430d5e740-cert\") pod \"ingress-canary-4tllr\" (UID: \"bfa0606a-1289-4a5d-8271-f17430d5e740\") " pod="openshift-ingress-canary/ingress-canary-4tllr" Sep 17 18:03:47.983623 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.983593 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aeee9793-7091-4047-b99c-871e17adafa0-config-volume\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:47.983623 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:47.983614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeee9793-7091-4047-b99c-871e17adafa0-tmp-dir\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.084758 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.084697 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aeee9793-7091-4047-b99c-871e17adafa0-config-volume\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.084758 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.084725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeee9793-7091-4047-b99c-871e17adafa0-tmp-dir\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.084758 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.084749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.084963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.084765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7gqj\" (UniqueName: \"kubernetes.io/projected/aeee9793-7091-4047-b99c-871e17adafa0-kube-api-access-q7gqj\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.084963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.084802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tk5j\" (UniqueName: \"kubernetes.io/projected/bfa0606a-1289-4a5d-8271-f17430d5e740-kube-api-access-8tk5j\") pod \"ingress-canary-4tllr\" (UID: \"bfa0606a-1289-4a5d-8271-f17430d5e740\") " pod="openshift-ingress-canary/ingress-canary-4tllr" Sep 17 18:03:48.084963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.084827 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa0606a-1289-4a5d-8271-f17430d5e740-cert\") pod \"ingress-canary-4tllr\" (UID: \"bfa0606a-1289-4a5d-8271-f17430d5e740\") " pod="openshift-ingress-canary/ingress-canary-4tllr" Sep 17 18:03:48.085079 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:48.084958 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:48.085079 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:48.085036 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls podName:aeee9793-7091-4047-b99c-871e17adafa0 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:48.585013062 +0000 UTC m=+39.648957016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls") pod "dns-default-42kg8" (UID: "aeee9793-7091-4047-b99c-871e17adafa0") : secret "dns-default-metrics-tls" not found Sep 17 18:03:48.085146 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.085087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeee9793-7091-4047-b99c-871e17adafa0-tmp-dir\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.085351 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.085335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aeee9793-7091-4047-b99c-871e17adafa0-config-volume\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.088858 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.088843 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bfa0606a-1289-4a5d-8271-f17430d5e740-cert\") pod \"ingress-canary-4tllr\" (UID: \"bfa0606a-1289-4a5d-8271-f17430d5e740\") " pod="openshift-ingress-canary/ingress-canary-4tllr" Sep 17 18:03:48.097898 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.097853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tk5j\" (UniqueName: \"kubernetes.io/projected/bfa0606a-1289-4a5d-8271-f17430d5e740-kube-api-access-8tk5j\") pod \"ingress-canary-4tllr\" (UID: \"bfa0606a-1289-4a5d-8271-f17430d5e740\") " pod="openshift-ingress-canary/ingress-canary-4tllr" Sep 17 18:03:48.098388 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.098365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7gqj\" (UniqueName: \"kubernetes.io/projected/aeee9793-7091-4047-b99c-871e17adafa0-kube-api-access-q7gqj\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.169614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.169587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4tllr" Sep 17 18:03:48.339700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.339639 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:03:48.343042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.343021 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 18:03:48.507695 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.507668 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4tllr"] Sep 17 18:03:48.512223 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:03:48.512195 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa0606a_1289_4a5d_8271_f17430d5e740.slice/crio-72ad0bfeaf01a1f0bcfeb741d461fa2654e616f35e9e8e1d1ed5ccd3821a50e5 WatchSource:0}: Error finding container 72ad0bfeaf01a1f0bcfeb741d461fa2654e616f35e9e8e1d1ed5ccd3821a50e5: Status 404 returned error can't find the container with id 72ad0bfeaf01a1f0bcfeb741d461fa2654e616f35e9e8e1d1ed5ccd3821a50e5 Sep 17 18:03:48.564153 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.564130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4tllr" event={"ID":"bfa0606a-1289-4a5d-8271-f17430d5e740","Type":"ContainerStarted","Data":"72ad0bfeaf01a1f0bcfeb741d461fa2654e616f35e9e8e1d1ed5ccd3821a50e5"} Sep 17 18:03:48.587766 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:48.587745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:48.587890 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:48.587855 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:48.587936 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:48.587918 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls podName:aeee9793-7091-4047-b99c-871e17adafa0 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:49.587902853 +0000 UTC m=+40.651846791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls") pod "dns-default-42kg8" (UID: "aeee9793-7091-4047-b99c-871e17adafa0") : secret "dns-default-metrics-tls" not found Sep 17 18:03:49.339649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:49.339614 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:03:49.339918 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:49.339890 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:03:49.342828 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:49.342807 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 18:03:49.343984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:49.343945 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qc24z\"" Sep 17 18:03:49.343984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:49.343947 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 18:03:49.344277 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:49.344263 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 18:03:49.344350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:49.344326 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2v7h2\"" Sep 17 18:03:49.595922 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:49.595837 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:49.596089 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:49.595992 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:49.596089 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:49.596064 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls podName:aeee9793-7091-4047-b99c-871e17adafa0 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:51.596046784 +0000 UTC m=+42.659990720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls") pod "dns-default-42kg8" (UID: "aeee9793-7091-4047-b99c-871e17adafa0") : secret "dns-default-metrics-tls" not found Sep 17 18:03:51.574764 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:51.574588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4tllr" event={"ID":"bfa0606a-1289-4a5d-8271-f17430d5e740","Type":"ContainerStarted","Data":"735d5749a2dc9f62808db504cd9dfb470396722237b22c0c4320e0cc48cc0080"} Sep 17 18:03:51.613221 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:51.613130 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:51.613337 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:51.613309 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:51.613377 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:51.613369 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls podName:aeee9793-7091-4047-b99c-871e17adafa0 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:55.613350371 +0000 UTC m=+46.677294323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls") pod "dns-default-42kg8" (UID: "aeee9793-7091-4047-b99c-871e17adafa0") : secret "dns-default-metrics-tls" not found Sep 17 18:03:55.637803 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:55.637772 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:03:55.638439 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:55.637922 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:03:55.638439 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:55.637989 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls podName:aeee9793-7091-4047-b99c-871e17adafa0 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:03.637968968 +0000 UTC m=+54.701912921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls") pod "dns-default-42kg8" (UID: "aeee9793-7091-4047-b99c-871e17adafa0") : secret "dns-default-metrics-tls" not found Sep 17 18:03:56.669512 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.669461 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4tllr" podStartSLOduration=6.839083997 podStartE2EDuration="9.669443324s" totalImagesPullingTime="2.830359327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:51.603894002 +0000 UTC m=+42.667837960" watchObservedRunningTime="2026-09-17 18:03:56.669443324 +0000 UTC m=+47.733387282" Sep 17 18:03:56.669930 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.669576 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp"] Sep 17 18:03:56.693819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.693796 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp"] Sep 17 18:03:56.693955 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.693906 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:56.697459 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.697433 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 17 18:03:56.697571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.697527 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tpr98\"" Sep 17 18:03:56.700737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.700711 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 18:03:56.701374 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.701349 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 18:03:56.702592 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.702549 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 17 18:03:56.844808 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.844781 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/998bbc1b-3f82-4e86-93f1-0dce59233c20-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:56.844950 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.844823 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wb8\" (UniqueName: \"kubernetes.io/projected/998bbc1b-3f82-4e86-93f1-0dce59233c20-kube-api-access-r7wb8\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:56.844950 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.844916 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:56.945555 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.945487 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/998bbc1b-3f82-4e86-93f1-0dce59233c20-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:56.945555 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.945534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wb8\" (UniqueName: \"kubernetes.io/projected/998bbc1b-3f82-4e86-93f1-0dce59233c20-kube-api-access-r7wb8\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:56.945706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.945690 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:56.945855 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:56.945839 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:03:56.945931 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:56.945917 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls podName:998bbc1b-3f82-4e86-93f1-0dce59233c20 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:57.445900721 +0000 UTC m=+48.509844671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xrmp" (UID: "998bbc1b-3f82-4e86-93f1-0dce59233c20") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:03:56.946210 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.946193 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/998bbc1b-3f82-4e86-93f1-0dce59233c20-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:56.958607 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:56.958588 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wb8\" (UniqueName: \"kubernetes.io/projected/998bbc1b-3f82-4e86-93f1-0dce59233c20-kube-api-access-r7wb8\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:57.449659 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:57.449636 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:57.449815 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:57.449746 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:03:57.449815 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:57.449811 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls podName:998bbc1b-3f82-4e86-93f1-0dce59233c20 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:58.44979512 +0000 UTC m=+49.513739069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xrmp" (UID: "998bbc1b-3f82-4e86-93f1-0dce59233c20") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:03:58.456796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:03:58.456759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:03:58.457299 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:58.456951 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:03:58.457299 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:03:58.457031 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls podName:998bbc1b-3f82-4e86-93f1-0dce59233c20 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:00.457011454 +0000 UTC m=+51.520955391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xrmp" (UID: "998bbc1b-3f82-4e86-93f1-0dce59233c20") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:04:00.471263 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.471227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:04:00.471630 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:00.471365 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:04:00.471630 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:00.471426 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls podName:998bbc1b-3f82-4e86-93f1-0dce59233c20 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.471410723 +0000 UTC m=+55.535354659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xrmp" (UID: "998bbc1b-3f82-4e86-93f1-0dce59233c20") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:04:00.559900 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.559850 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5xfxv"] Sep 17 18:04:00.562017 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.562002 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.565333 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.565310 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 18:04:00.566345 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.566329 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 18:04:00.566429 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.566375 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-87jlz\"" Sep 17 18:04:00.566429 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.566394 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 18:04:00.566538 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.566375 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 18:04:00.602624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.602602 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xfxv"] Sep 17 18:04:00.671963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.671939 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8342fcb-842b-4f54-8f5b-7d34215c9091-data-volume\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.672066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.671967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99fs\" (UniqueName: \"kubernetes.io/projected/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-api-access-d99fs\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.672066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.672024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8342fcb-842b-4f54-8f5b-7d34215c9091-crio-socket\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.672066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.672049 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.672174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.672072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.728403 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.728348 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx"] Sep 17 18:04:00.730141 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.730126 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx" Sep 17 18:04:00.734505 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.734490 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 17 18:04:00.734758 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.734740 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:04:00.736626 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.736611 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-mxq6w\"" Sep 17 18:04:00.751576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.751554 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6"] Sep 17 18:04:00.756483 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.756465 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:00.766586 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.766566 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx"] Sep 17 18:04:00.767406 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.767384 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 18:04:00.767770 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.767753 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 18:04:00.767845 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.767774 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:04:00.768019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.767999 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5bvvb\"" Sep 17 18:04:00.772984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.772966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.773062 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.773024 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8342fcb-842b-4f54-8f5b-7d34215c9091-data-volume\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.773062 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.773048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99fs\" (UniqueName: \"kubernetes.io/projected/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-api-access-d99fs\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.773130 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:00.773111 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:00.773173 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:00.773162 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.273147286 +0000 UTC m=+52.337091228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:00.773249 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.773110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8342fcb-842b-4f54-8f5b-7d34215c9091-crio-socket\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.773360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.773264 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e8342fcb-842b-4f54-8f5b-7d34215c9091-crio-socket\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.773360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.773279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.773360 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:00.773329 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:00.773360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.773349 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8342fcb-842b-4f54-8f5b-7d34215c9091-data-volume\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.773528 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:00.773383 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.273371555 +0000 UTC m=+52.337315506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : configmap "kube-rbac-proxy" not found Sep 17 18:04:00.781410 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.781392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6"] Sep 17 18:04:00.805220 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.805195 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99fs\" (UniqueName: \"kubernetes.io/projected/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-api-access-d99fs\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:00.873625 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.873600 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rqj\" (UniqueName: \"kubernetes.io/projected/1c794bd0-5e63-4cb0-aaef-8c909d75ea1a-kube-api-access-g8rqj\") pod \"volume-data-source-validator-5c4db8bdf9-bj4gx\" (UID: \"1c794bd0-5e63-4cb0-aaef-8c909d75ea1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx" Sep 17 18:04:00.873625 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.873627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jv2\" (UniqueName: \"kubernetes.io/projected/89b3f083-c374-4bbc-b3f2-dda2a50977e4-kube-api-access-w7jv2\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:00.873776 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.873756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:00.974167 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.974144 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:00.974258 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.974182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rqj\" (UniqueName: \"kubernetes.io/projected/1c794bd0-5e63-4cb0-aaef-8c909d75ea1a-kube-api-access-g8rqj\") pod \"volume-data-source-validator-5c4db8bdf9-bj4gx\" (UID: \"1c794bd0-5e63-4cb0-aaef-8c909d75ea1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx" Sep 17 18:04:00.974258 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.974198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jv2\" (UniqueName: \"kubernetes.io/projected/89b3f083-c374-4bbc-b3f2-dda2a50977e4-kube-api-access-w7jv2\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:00.974322 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:00.974290 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:04:00.974402 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:00.974347 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls podName:89b3f083-c374-4bbc-b3f2-dda2a50977e4 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.47433344 +0000 UTC m=+52.538277391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-7lhd6" (UID: "89b3f083-c374-4bbc-b3f2-dda2a50977e4") : secret "samples-operator-tls" not found Sep 17 18:04:00.989685 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.989627 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jv2\" (UniqueName: \"kubernetes.io/projected/89b3f083-c374-4bbc-b3f2-dda2a50977e4-kube-api-access-w7jv2\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:00.990081 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:00.990064 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rqj\" (UniqueName: \"kubernetes.io/projected/1c794bd0-5e63-4cb0-aaef-8c909d75ea1a-kube-api-access-g8rqj\") pod \"volume-data-source-validator-5c4db8bdf9-bj4gx\" (UID: \"1c794bd0-5e63-4cb0-aaef-8c909d75ea1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx" Sep 17 18:04:01.038873 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.038827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx" Sep 17 18:04:01.277296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.277206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:01.277296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.277253 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:01.277513 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:01.277362 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:01.277513 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:01.277423 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:02.277406216 +0000 UTC m=+53.341350157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:01.277513 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:01.277448 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:01.277662 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:01.277534 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:02.277519282 +0000 UTC m=+53.341463240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : configmap "kube-rbac-proxy" not found Sep 17 18:04:01.312520 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.312493 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx"] Sep 17 18:04:01.316831 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:01.316804 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c794bd0_5e63_4cb0_aaef_8c909d75ea1a.slice/crio-95e02412e58b99fc3ff759e6c9176113fd8bcb0cf91b60cb85f9d9b5fea6d914 WatchSource:0}: Error finding container 95e02412e58b99fc3ff759e6c9176113fd8bcb0cf91b60cb85f9d9b5fea6d914: Status 404 returned error can't find the container with id 95e02412e58b99fc3ff759e6c9176113fd8bcb0cf91b60cb85f9d9b5fea6d914 Sep 17 18:04:01.479301 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.479268 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:01.479632 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:01.479370 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:04:01.479632 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:01.479421 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls podName:89b3f083-c374-4bbc-b3f2-dda2a50977e4 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:02.479407984 +0000 UTC m=+53.543351919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-7lhd6" (UID: "89b3f083-c374-4bbc-b3f2-dda2a50977e4") : secret "samples-operator-tls" not found Sep 17 18:04:01.598633 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.598556 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx" event={"ID":"1c794bd0-5e63-4cb0-aaef-8c909d75ea1a","Type":"ContainerStarted","Data":"95e02412e58b99fc3ff759e6c9176113fd8bcb0cf91b60cb85f9d9b5fea6d914"} Sep 17 18:04:01.649624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.649598 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4fqtn"] Sep 17 18:04:01.651977 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.651963 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.655529 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.655510 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 17 18:04:01.656197 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.656178 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 17 18:04:01.656290 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.656208 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:04:01.656290 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.656236 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-72p77\"" Sep 17 18:04:01.657341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.657326 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 17 18:04:01.662379 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.662232 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 17 18:04:01.673393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.673366 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4fqtn"] Sep 17 18:04:01.782717 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.782682 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/576187ca-ff17-474c-b9e5-c8b17ac8e982-serving-cert\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.782717 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.782716 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/576187ca-ff17-474c-b9e5-c8b17ac8e982-trusted-ca\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.782929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.782734 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/576187ca-ff17-474c-b9e5-c8b17ac8e982-config\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.782929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.782749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jb9\" (UniqueName: \"kubernetes.io/projected/576187ca-ff17-474c-b9e5-c8b17ac8e982-kube-api-access-c4jb9\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.883472 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.883395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/576187ca-ff17-474c-b9e5-c8b17ac8e982-serving-cert\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.883472 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.883439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/576187ca-ff17-474c-b9e5-c8b17ac8e982-trusted-ca\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.883700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.883678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/576187ca-ff17-474c-b9e5-c8b17ac8e982-config\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.883759 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.883725 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jb9\" (UniqueName: \"kubernetes.io/projected/576187ca-ff17-474c-b9e5-c8b17ac8e982-kube-api-access-c4jb9\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.884369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.884346 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/576187ca-ff17-474c-b9e5-c8b17ac8e982-config\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.884531 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.884515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/576187ca-ff17-474c-b9e5-c8b17ac8e982-trusted-ca\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.886135 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.886112 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/576187ca-ff17-474c-b9e5-c8b17ac8e982-serving-cert\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.894430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.894395 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jb9\" (UniqueName: \"kubernetes.io/projected/576187ca-ff17-474c-b9e5-c8b17ac8e982-kube-api-access-c4jb9\") pod \"console-operator-fd4bb46c-4fqtn\" (UID: \"576187ca-ff17-474c-b9e5-c8b17ac8e982\") " pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:01.966786 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:01.966755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:02.255759 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:02.255730 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-4fqtn"] Sep 17 18:04:02.286227 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:02.286198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:02.286227 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:02.286237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:02.286452 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:02.286356 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:02.286452 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:02.286378 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:02.286452 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:02.286430 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.286415872 +0000 UTC m=+55.350359836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:02.286452 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:02.286446 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.286438417 +0000 UTC m=+55.350382353 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : configmap "kube-rbac-proxy" not found Sep 17 18:04:02.419684 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:02.419655 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576187ca_ff17_474c_b9e5_c8b17ac8e982.slice/crio-83c7e500897618bdc2c762ebfef90d29f38bfdd26e8afd0138fc5243b9d262cf WatchSource:0}: Error finding container 83c7e500897618bdc2c762ebfef90d29f38bfdd26e8afd0138fc5243b9d262cf: Status 404 returned error can't find the container with id 83c7e500897618bdc2c762ebfef90d29f38bfdd26e8afd0138fc5243b9d262cf Sep 17 18:04:02.430895 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:02.430858 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-spg6t_23d3772b-6060-4903-be8a-49a0704c9a3e/dns-node-resolver/0.log" Sep 17 18:04:02.488044 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:02.488018 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:02.488321 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:02.488159 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:04:02.488321 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:02.488211 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls podName:89b3f083-c374-4bbc-b3f2-dda2a50977e4 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.488197835 +0000 UTC m=+55.552141771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-7lhd6" (UID: "89b3f083-c374-4bbc-b3f2-dda2a50977e4") : secret "samples-operator-tls" not found Sep 17 18:04:02.602929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:02.602830 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx" event={"ID":"1c794bd0-5e63-4cb0-aaef-8c909d75ea1a","Type":"ContainerStarted","Data":"bf5620f4093a6d88e83e9e5da7236b2afa6dedc92f0763306b2852b8d24f3d3e"} Sep 17 18:04:02.604027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:02.604004 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" event={"ID":"576187ca-ff17-474c-b9e5-c8b17ac8e982","Type":"ContainerStarted","Data":"83c7e500897618bdc2c762ebfef90d29f38bfdd26e8afd0138fc5243b9d262cf"} Sep 17 18:04:02.623608 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:02.623569 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-bj4gx" podStartSLOduration=1.500989586 podStartE2EDuration="2.623554709s" totalImagesPullingTime="1.122565123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:02.622888434 +0000 UTC m=+53.686832383" watchObservedRunningTime="2026-09-17 18:04:02.623554709 +0000 UTC m=+53.687498668" Sep 17 18:04:03.233653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:03.233627 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ktpwl_7398a155-5335-448b-b8b5-c86efaf5e6b6/node-ca/0.log" Sep 17 18:04:03.698297 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:03.698257 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:04:03.698741 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:03.698424 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:04:03.698741 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:03.698504 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls podName:aeee9793-7091-4047-b99c-871e17adafa0 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:19.698484089 +0000 UTC m=+70.762428027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls") pod "dns-default-42kg8" (UID: "aeee9793-7091-4047-b99c-871e17adafa0") : secret "dns-default-metrics-tls" not found Sep 17 18:04:03.832784 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:03.832749 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4tllr_bfa0606a-1289-4a5d-8271-f17430d5e740/serve-healthcheck-canary/0.log" Sep 17 18:04:04.302707 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:04.302672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:04.302707 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:04.302713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:04.302935 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:04.302802 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:04.302935 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:04.302896 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:08.302877052 +0000 UTC m=+59.366821003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : configmap "kube-rbac-proxy" not found Sep 17 18:04:04.303007 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:04.302949 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:04.303007 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:04.302992 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:08.302980664 +0000 UTC m=+59.366924617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:04.504834 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:04.504797 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:04.505028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:04.504919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:04:04.505028 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:04.504978 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:04:04.505137 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:04.505045 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls podName:89b3f083-c374-4bbc-b3f2-dda2a50977e4 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:08.505030484 +0000 UTC m=+59.568974445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-7lhd6" (UID: "89b3f083-c374-4bbc-b3f2-dda2a50977e4") : secret "samples-operator-tls" not found Sep 17 18:04:04.505137 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:04.505056 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:04:04.505137 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:04.505111 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls podName:998bbc1b-3f82-4e86-93f1-0dce59233c20 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:12.505091884 +0000 UTC m=+63.569035842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xrmp" (UID: "998bbc1b-3f82-4e86-93f1-0dce59233c20") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:04:05.613284 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.613256 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/0.log" Sep 17 18:04:05.613650 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.613299 2582 generic.go:356] "Generic (PLEG): container finished" podID="576187ca-ff17-474c-b9e5-c8b17ac8e982" containerID="938529806b63ff3160dd1ea337e42ee916edd01c508290d5f15d492db60ba6f3" exitCode=255 Sep 17 18:04:05.613650 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.613337 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" event={"ID":"576187ca-ff17-474c-b9e5-c8b17ac8e982","Type":"ContainerDied","Data":"938529806b63ff3160dd1ea337e42ee916edd01c508290d5f15d492db60ba6f3"} Sep 17 18:04:05.613719 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.613664 2582 scope.go:117] "RemoveContainer" containerID="938529806b63ff3160dd1ea337e42ee916edd01c508290d5f15d492db60ba6f3" Sep 17 18:04:05.648332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.648299 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6"] Sep 17 18:04:05.678704 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.678676 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6"] Sep 17 18:04:05.678840 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.678779 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.681849 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.681818 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vnzcj\"" Sep 17 18:04:05.681849 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.681816 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:04:05.682022 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.681899 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 18:04:05.682613 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.682598 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 18:04:05.682764 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.682748 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 18:04:05.815508 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.815477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/661c1262-8856-4aa9-be54-f3a2594cee26-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.815508 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.815513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/661c1262-8856-4aa9-be54-f3a2594cee26-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.815653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.815539 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plw2g\" (UniqueName: \"kubernetes.io/projected/661c1262-8856-4aa9-be54-f3a2594cee26-kube-api-access-plw2g\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.916298 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.916266 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/661c1262-8856-4aa9-be54-f3a2594cee26-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.916298 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.916300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/661c1262-8856-4aa9-be54-f3a2594cee26-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.916512 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.916491 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plw2g\" (UniqueName: \"kubernetes.io/projected/661c1262-8856-4aa9-be54-f3a2594cee26-kube-api-access-plw2g\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.916768 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.916751 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/661c1262-8856-4aa9-be54-f3a2594cee26-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.918372 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.918350 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/661c1262-8856-4aa9-be54-f3a2594cee26-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.925535 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.925514 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plw2g\" (UniqueName: \"kubernetes.io/projected/661c1262-8856-4aa9-be54-f3a2594cee26-kube-api-access-plw2g\") pod \"kube-storage-version-migrator-operator-55889f7b47-frch6\" (UID: \"661c1262-8856-4aa9-be54-f3a2594cee26\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:05.987577 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:05.987554 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" Sep 17 18:04:06.275920 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:06.275838 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6"] Sep 17 18:04:06.283325 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:06.283284 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661c1262_8856_4aa9_be54_f3a2594cee26.slice/crio-8c8e58a34dc87d8fe6941764bd005c6d4eb2dc0964ec6de414f0dfe419a1c18a WatchSource:0}: Error finding container 8c8e58a34dc87d8fe6941764bd005c6d4eb2dc0964ec6de414f0dfe419a1c18a: Status 404 returned error can't find the container with id 8c8e58a34dc87d8fe6941764bd005c6d4eb2dc0964ec6de414f0dfe419a1c18a Sep 17 18:04:06.617456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:06.617389 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:04:06.617905 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:06.617770 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/0.log" Sep 17 18:04:06.617905 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:06.617802 2582 generic.go:356] "Generic (PLEG): container finished" podID="576187ca-ff17-474c-b9e5-c8b17ac8e982" containerID="24d64bde48d54b6ffcea0383f03d9c506300b9d12419a99d93034fb2090701c5" exitCode=255 Sep 17 18:04:06.618030 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:06.617896 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" event={"ID":"576187ca-ff17-474c-b9e5-c8b17ac8e982","Type":"ContainerDied","Data":"24d64bde48d54b6ffcea0383f03d9c506300b9d12419a99d93034fb2090701c5"} Sep 17 18:04:06.618030 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:06.617957 2582 scope.go:117] "RemoveContainer" containerID="938529806b63ff3160dd1ea337e42ee916edd01c508290d5f15d492db60ba6f3" Sep 17 18:04:06.618180 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:06.618158 2582 scope.go:117] "RemoveContainer" containerID="24d64bde48d54b6ffcea0383f03d9c506300b9d12419a99d93034fb2090701c5" Sep 17 18:04:06.618359 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:06.618341 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4fqtn_openshift-console-operator(576187ca-ff17-474c-b9e5-c8b17ac8e982)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" podUID="576187ca-ff17-474c-b9e5-c8b17ac8e982" Sep 17 18:04:06.619341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:06.619242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" event={"ID":"661c1262-8856-4aa9-be54-f3a2594cee26","Type":"ContainerStarted","Data":"8c8e58a34dc87d8fe6941764bd005c6d4eb2dc0964ec6de414f0dfe419a1c18a"} Sep 17 18:04:07.624420 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:07.624394 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:04:07.624887 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:07.624788 2582 scope.go:117] "RemoveContainer" containerID="24d64bde48d54b6ffcea0383f03d9c506300b9d12419a99d93034fb2090701c5" Sep 17 18:04:07.625023 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:07.625002 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4fqtn_openshift-console-operator(576187ca-ff17-474c-b9e5-c8b17ac8e982)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" podUID="576187ca-ff17-474c-b9e5-c8b17ac8e982" Sep 17 18:04:08.333124 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.333098 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:08.333287 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.333134 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:08.333287 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.333225 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 18:04:08.333287 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.333243 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:08.333287 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.333283 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.333270564 +0000 UTC m=+67.397214500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:08.333423 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.333297 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.333289878 +0000 UTC m=+67.397233814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : configmap "kube-rbac-proxy" not found Sep 17 18:04:08.534789 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.534752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:08.534973 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.534895 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:04:08.534973 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.534965 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls podName:89b3f083-c374-4bbc-b3f2-dda2a50977e4 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.534945882 +0000 UTC m=+67.598889840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-7lhd6" (UID: "89b3f083-c374-4bbc-b3f2-dda2a50977e4") : secret "samples-operator-tls" not found Sep 17 18:04:08.783002 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.782962 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6679f8698c-888bv"] Sep 17 18:04:08.785449 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.785426 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.789838 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.789611 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 18:04:08.789838 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.789733 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 18:04:08.793510 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.793485 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 18:04:08.793630 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.793563 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kgd2r\"" Sep 17 18:04:08.797706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.797685 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 18:04:08.806452 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.806415 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6679f8698c-888bv"] Sep 17 18:04:08.837220 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.837184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-trusted-ca\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.837408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.837265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.837408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.837355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-bound-sa-token\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.837408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.837392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-installation-pull-secrets\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.837575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.837421 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152119e1-0219-4f51-97ba-6e06e41b8487-ca-trust-extracted\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.837575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.837448 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5xg\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-kube-api-access-bm5xg\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.837575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.837486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-registry-certificates\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.837704 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.837612 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-image-registry-private-configuration\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.938833 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.938809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-image-registry-private-configuration\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.938942 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.938889 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-trusted-ca\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.938942 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.938922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.939038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.938943 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-bound-sa-token\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.939038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.938964 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-installation-pull-secrets\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.939038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.938990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152119e1-0219-4f51-97ba-6e06e41b8487-ca-trust-extracted\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.939186 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.939053 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:04:08.939186 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.939071 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6679f8698c-888bv: secret "image-registry-tls" not found Sep 17 18:04:08.939186 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:08.939132 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls podName:152119e1-0219-4f51-97ba-6e06e41b8487 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:09.439112995 +0000 UTC m=+60.503056954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls") pod "image-registry-6679f8698c-888bv" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487") : secret "image-registry-tls" not found Sep 17 18:04:08.939341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.939212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5xg\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-kube-api-access-bm5xg\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.939341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.939273 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-registry-certificates\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.939341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.939321 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152119e1-0219-4f51-97ba-6e06e41b8487-ca-trust-extracted\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.939928 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.939851 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-registry-certificates\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.940173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.940154 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-trusted-ca\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.941207 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.941177 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-image-registry-private-configuration\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.941354 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.941336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-installation-pull-secrets\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.951971 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.951945 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-bound-sa-token\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:08.952051 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:08.952032 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5xg\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-kube-api-access-bm5xg\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:09.443821 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:09.443782 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:09.444045 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:09.443923 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:04:09.444045 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:09.443936 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6679f8698c-888bv: secret "image-registry-tls" not found Sep 17 18:04:09.444045 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:09.443988 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls podName:152119e1-0219-4f51-97ba-6e06e41b8487 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:10.4439744 +0000 UTC m=+61.507918336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls") pod "image-registry-6679f8698c-888bv" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487") : secret "image-registry-tls" not found Sep 17 18:04:09.631021 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:09.630987 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" event={"ID":"661c1262-8856-4aa9-be54-f3a2594cee26","Type":"ContainerStarted","Data":"f425edb87baac03f7833e3acb7e1496183d5165bac11e52fb95bf271835b4786"} Sep 17 18:04:09.657976 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:09.657925 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" podStartSLOduration=2.018761892 podStartE2EDuration="4.657908937s" totalImagesPullingTime="2.639147045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:09.657601033 +0000 UTC m=+60.721545090" watchObservedRunningTime="2026-09-17 18:04:09.657908937 +0000 UTC m=+60.721852895" Sep 17 18:04:10.450045 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:10.450002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:10.450507 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:10.450193 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:04:10.450507 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:10.450222 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6679f8698c-888bv: secret "image-registry-tls" not found Sep 17 18:04:10.450507 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:10.450296 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls podName:152119e1-0219-4f51-97ba-6e06e41b8487 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:12.450273972 +0000 UTC m=+63.514217930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls") pod "image-registry-6679f8698c-888bv" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487") : secret "image-registry-tls" not found Sep 17 18:04:10.547182 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:10.547150 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kqs7r" Sep 17 18:04:11.967719 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:11.967683 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:11.968146 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:11.967747 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:11.968188 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:11.968168 2582 scope.go:117] "RemoveContainer" containerID="24d64bde48d54b6ffcea0383f03d9c506300b9d12419a99d93034fb2090701c5" Sep 17 18:04:11.968347 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:11.968322 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-4fqtn_openshift-console-operator(576187ca-ff17-474c-b9e5-c8b17ac8e982)\"" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" podUID="576187ca-ff17-474c-b9e5-c8b17ac8e982" Sep 17 18:04:12.468804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:12.468769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:12.468992 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:12.468924 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:04:12.468992 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:12.468943 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6679f8698c-888bv: secret "image-registry-tls" not found Sep 17 18:04:12.469068 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:12.468996 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls podName:152119e1-0219-4f51-97ba-6e06e41b8487 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.46898038 +0000 UTC m=+67.532924329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls") pod "image-registry-6679f8698c-888bv" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487") : secret "image-registry-tls" not found Sep 17 18:04:12.570202 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:12.570162 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:04:12.570357 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:12.570312 2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 17 18:04:12.570396 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:12.570379 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls podName:998bbc1b-3f82-4e86-93f1-0dce59233c20 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:28.570363918 +0000 UTC m=+79.634307858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-4xrmp" (UID: "998bbc1b-3f82-4e86-93f1-0dce59233c20") : secret "cluster-monitoring-operator-tls" not found Sep 17 18:04:16.097530 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.097476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:04:16.097530 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.097542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:04:16.101259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.101240 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 18:04:16.101656 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.101641 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 18:04:16.107841 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.107820 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 18:04:16.107929 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.107912 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs podName:9dd86bb7-9b68-4957-87ba-905e2ffe4948 nodeName:}" failed. No retries permitted until 2026-09-17 18:05:20.107894068 +0000 UTC m=+131.171838004 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs") pod "network-metrics-daemon-ctrs8" (UID: "9dd86bb7-9b68-4957-87ba-905e2ffe4948") : secret "metrics-daemon-secret" not found Sep 17 18:04:16.109955 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.109937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/912b1cfc-fb70-47aa-9558-f12d1d3f4416-original-pull-secret\") pod \"global-pull-secret-syncer-tzwqq\" (UID: \"912b1cfc-fb70-47aa-9558-f12d1d3f4416\") " pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:04:16.198846 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.198799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:04:16.201796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.201781 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 18:04:16.211761 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.211746 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 18:04:16.221690 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.221659 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx72z\" (UniqueName: \"kubernetes.io/projected/e8f6ca4e-3cc1-4d22-820e-afc5cdf98276-kube-api-access-cx72z\") pod \"network-check-target-wrnpc\" (UID: \"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276\") " pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:04:16.248716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.248694 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tzwqq" Sep 17 18:04:16.354589 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.354512 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2v7h2\"" Sep 17 18:04:16.361856 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.361836 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:04:16.401218 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.401188 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:16.401394 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.401225 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:16.401394 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.401356 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 18:04:16.401510 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.401404 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls podName:e8342fcb-842b-4f54-8f5b-7d34215c9091 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:32.401389114 +0000 UTC m=+83.465333063 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xfxv" (UID: "e8342fcb-842b-4f54-8f5b-7d34215c9091") : secret "insights-runtime-extractor-tls" not found Sep 17 18:04:16.401773 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.401749 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e8342fcb-842b-4f54-8f5b-7d34215c9091-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:16.502303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.502265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:16.502454 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.502435 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 18:04:16.502524 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.502457 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6679f8698c-888bv: secret "image-registry-tls" not found Sep 17 18:04:16.502585 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.502529 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls podName:152119e1-0219-4f51-97ba-6e06e41b8487 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:24.502506527 +0000 UTC m=+75.566450478 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls") pod "image-registry-6679f8698c-888bv" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487") : secret "image-registry-tls" not found Sep 17 18:04:16.519424 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.519399 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tzwqq"] Sep 17 18:04:16.522504 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:16.522484 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912b1cfc_fb70_47aa_9558_f12d1d3f4416.slice/crio-83a9aef4aa245f6323131b2013a4f3acd63f1c556d9cde25e47734db826d211e WatchSource:0}: Error finding container 83a9aef4aa245f6323131b2013a4f3acd63f1c556d9cde25e47734db826d211e: Status 404 returned error can't find the container with id 83a9aef4aa245f6323131b2013a4f3acd63f1c556d9cde25e47734db826d211e Sep 17 18:04:16.602989 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.602957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:16.603145 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.603092 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 18:04:16.603145 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:16.603145 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls podName:89b3f083-c374-4bbc-b3f2-dda2a50977e4 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:32.603132447 +0000 UTC m=+83.667076383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-7lhd6" (UID: "89b3f083-c374-4bbc-b3f2-dda2a50977e4") : secret "samples-operator-tls" not found Sep 17 18:04:16.632456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.632385 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wrnpc"] Sep 17 18:04:16.636234 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:16.636211 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f6ca4e_3cc1_4d22_820e_afc5cdf98276.slice/crio-3717b45530aadda9da5588c2a0c633266fe73e3c7b0ffe31c4caee97e19d3078 WatchSource:0}: Error finding container 3717b45530aadda9da5588c2a0c633266fe73e3c7b0ffe31c4caee97e19d3078: Status 404 returned error can't find the container with id 3717b45530aadda9da5588c2a0c633266fe73e3c7b0ffe31c4caee97e19d3078 Sep 17 18:04:16.649361 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.649334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tzwqq" event={"ID":"912b1cfc-fb70-47aa-9558-f12d1d3f4416","Type":"ContainerStarted","Data":"83a9aef4aa245f6323131b2013a4f3acd63f1c556d9cde25e47734db826d211e"} Sep 17 18:04:16.650405 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:16.650387 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wrnpc" event={"ID":"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276","Type":"ContainerStarted","Data":"3717b45530aadda9da5588c2a0c633266fe73e3c7b0ffe31c4caee97e19d3078"} Sep 17 18:04:19.732995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:19.732947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:04:19.733550 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:19.733143 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 18:04:19.733550 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:19.733219 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls podName:aeee9793-7091-4047-b99c-871e17adafa0 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:51.733200937 +0000 UTC m=+102.797144873 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls") pod "dns-default-42kg8" (UID: "aeee9793-7091-4047-b99c-871e17adafa0") : secret "dns-default-metrics-tls" not found Sep 17 18:04:21.667700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:21.667663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wrnpc" event={"ID":"e8f6ca4e-3cc1-4d22-820e-afc5cdf98276","Type":"ContainerStarted","Data":"98599899e7f3db1041b57aa379d2bd54c8901cf8d6319704076c9df47c02bae4"} Sep 17 18:04:21.668190 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:21.667723 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:04:21.669171 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:21.669148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tzwqq" event={"ID":"912b1cfc-fb70-47aa-9558-f12d1d3f4416","Type":"ContainerStarted","Data":"de20b333d11e5b6ae0f23a6220bfbe017da31d42317f5e296b6cdc2f010b7443"} Sep 17 18:04:21.713839 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:21.713793 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wrnpc" podStartSLOduration=66.033042341 podStartE2EDuration="1m10.71377952s" totalImagesPullingTime="4.680737179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:21.71258252 +0000 UTC m=+72.776526478" watchObservedRunningTime="2026-09-17 18:04:21.71377952 +0000 UTC m=+72.777723478" Sep 17 18:04:21.732757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:21.732715 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tzwqq" podStartSLOduration=65.934461767 podStartE2EDuration="1m10.732701542s" totalImagesPullingTime="4.798239775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:21.731906685 +0000 UTC m=+72.795850642" watchObservedRunningTime="2026-09-17 18:04:21.732701542 +0000 UTC m=+72.796645499" Sep 17 18:04:24.573069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:24.572948 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:24.575258 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:24.575238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"image-registry-6679f8698c-888bv\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:24.703260 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:24.703232 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kgd2r\"" Sep 17 18:04:24.711554 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:24.711534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:24.990470 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:24.990439 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6679f8698c-888bv"] Sep 17 18:04:24.993196 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:24.993165 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152119e1_0219_4f51_97ba_6e06e41b8487.slice/crio-e1ef7ffa7c6e7566a601562b81fc687cb04b0b463d3f9261b14bce8a2b5e4a5d WatchSource:0}: Error finding container e1ef7ffa7c6e7566a601562b81fc687cb04b0b463d3f9261b14bce8a2b5e4a5d: Status 404 returned error can't find the container with id e1ef7ffa7c6e7566a601562b81fc687cb04b0b463d3f9261b14bce8a2b5e4a5d Sep 17 18:04:25.683709 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:25.683666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6679f8698c-888bv" event={"ID":"152119e1-0219-4f51-97ba-6e06e41b8487","Type":"ContainerStarted","Data":"42579c28f6825c2632c2bc0276141dec2fb06494a2a82ee58a8735b568d70c1c"} Sep 17 18:04:25.684254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:25.683725 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6679f8698c-888bv" event={"ID":"152119e1-0219-4f51-97ba-6e06e41b8487","Type":"ContainerStarted","Data":"e1ef7ffa7c6e7566a601562b81fc687cb04b0b463d3f9261b14bce8a2b5e4a5d"} Sep 17 18:04:25.684254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:25.683808 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:25.709491 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:25.709443 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6679f8698c-888bv" podStartSLOduration=17.709428678 podStartE2EDuration="17.709428678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:25.708742985 +0000 UTC m=+76.772686942" watchObservedRunningTime="2026-09-17 18:04:25.709428678 +0000 UTC m=+76.773372635" Sep 17 18:04:26.339220 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:26.339188 2582 scope.go:117] "RemoveContainer" containerID="24d64bde48d54b6ffcea0383f03d9c506300b9d12419a99d93034fb2090701c5" Sep 17 18:04:26.693820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:26.693789 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:04:26.694281 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:26.693923 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" event={"ID":"576187ca-ff17-474c-b9e5-c8b17ac8e982","Type":"ContainerStarted","Data":"c4c437e54ba8cf2cd8a63a432f1eb9ce011fea3cb50acd505c40279670879ced"} Sep 17 18:04:26.694377 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:26.694308 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:27.214417 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:27.214186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" Sep 17 18:04:27.241171 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:27.241112 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-4fqtn" podStartSLOduration=24.132451098 podStartE2EDuration="26.241095439s" totalImagesPullingTime="2.108644341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:26.718812605 +0000 UTC m=+77.782756564" watchObservedRunningTime="2026-09-17 18:04:27.241095439 +0000 UTC m=+78.305039398" Sep 17 18:04:28.606040 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:28.605997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:04:28.608285 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:28.608262 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/998bbc1b-3f82-4e86-93f1-0dce59233c20-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-4xrmp\" (UID: \"998bbc1b-3f82-4e86-93f1-0dce59233c20\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:04:28.806546 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:28.806518 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tpr98\"" Sep 17 18:04:28.814488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:28.814467 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" Sep 17 18:04:29.089244 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:29.089213 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp"] Sep 17 18:04:29.092639 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:29.092609 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998bbc1b_3f82_4e86_93f1_0dce59233c20.slice/crio-0a6221284b83f11e059b79b3427355748b939108ea737393a32f1ac7ec68b527 WatchSource:0}: Error finding container 0a6221284b83f11e059b79b3427355748b939108ea737393a32f1ac7ec68b527: Status 404 returned error can't find the container with id 0a6221284b83f11e059b79b3427355748b939108ea737393a32f1ac7ec68b527 Sep 17 18:04:29.706747 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:29.706712 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" event={"ID":"998bbc1b-3f82-4e86-93f1-0dce59233c20","Type":"ContainerStarted","Data":"0a6221284b83f11e059b79b3427355748b939108ea737393a32f1ac7ec68b527"} Sep 17 18:04:30.711526 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:30.711457 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" event={"ID":"998bbc1b-3f82-4e86-93f1-0dce59233c20","Type":"ContainerStarted","Data":"10b58f12d0801e8feb644940351d7485f3787bc91d35550b42a40ffbe9d5dbef"} Sep 17 18:04:30.734049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:30.733994 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-4xrmp" podStartSLOduration=33.181306087 podStartE2EDuration="34.733973584s" totalImagesPullingTime="1.552667497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:30.732553986 +0000 UTC m=+81.796497946" watchObservedRunningTime="2026-09-17 18:04:30.733973584 +0000 UTC m=+81.797917552" Sep 17 18:04:32.439897 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.439850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:32.442148 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.442125 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e8342fcb-842b-4f54-8f5b-7d34215c9091-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xfxv\" (UID: \"e8342fcb-842b-4f54-8f5b-7d34215c9091\") " pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:32.640755 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.640726 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:32.643027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.642998 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89b3f083-c374-4bbc-b3f2-dda2a50977e4-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-7lhd6\" (UID: \"89b3f083-c374-4bbc-b3f2-dda2a50977e4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:32.673305 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.673285 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-87jlz\"" Sep 17 18:04:32.681800 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.681781 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xfxv" Sep 17 18:04:32.866909 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.866881 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5bvvb\"" Sep 17 18:04:32.875076 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.875060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" Sep 17 18:04:32.957992 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:32.957927 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xfxv"] Sep 17 18:04:32.961164 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:32.961140 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8342fcb_842b_4f54_8f5b_7d34215c9091.slice/crio-741f29c9beda3a9d79e030873164cae1a7e4851638f673ebbfba2fb32bcf1edd WatchSource:0}: Error finding container 741f29c9beda3a9d79e030873164cae1a7e4851638f673ebbfba2fb32bcf1edd: Status 404 returned error can't find the container with id 741f29c9beda3a9d79e030873164cae1a7e4851638f673ebbfba2fb32bcf1edd Sep 17 18:04:33.146242 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:33.146202 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6"] Sep 17 18:04:33.722950 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:33.722916 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xfxv" event={"ID":"e8342fcb-842b-4f54-8f5b-7d34215c9091","Type":"ContainerStarted","Data":"7dc8d4704441c98e67f51c54ee2a76f8762d97c931bd8a29622e069f8369530a"} Sep 17 18:04:33.723325 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:33.722982 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xfxv" event={"ID":"e8342fcb-842b-4f54-8f5b-7d34215c9091","Type":"ContainerStarted","Data":"741f29c9beda3a9d79e030873164cae1a7e4851638f673ebbfba2fb32bcf1edd"} Sep 17 18:04:33.724031 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:33.724011 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" event={"ID":"89b3f083-c374-4bbc-b3f2-dda2a50977e4","Type":"ContainerStarted","Data":"d21b1902901562b9d3fdcb9ab7c76376fe263edf703fd11fa94d33c5040c6c23"} Sep 17 18:04:34.729530 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:34.729497 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xfxv" event={"ID":"e8342fcb-842b-4f54-8f5b-7d34215c9091","Type":"ContainerStarted","Data":"8a9d0ef604bc0d1b20b0f40808fb225349918f89b12f9addc02c299845194892"} Sep 17 18:04:35.736048 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:35.735951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xfxv" event={"ID":"e8342fcb-842b-4f54-8f5b-7d34215c9091","Type":"ContainerStarted","Data":"8b0fa16305b3e78168c49cfd369cb6e650ee34abae1df0e992fb61383811fb15"} Sep 17 18:04:35.738662 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:35.738632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" event={"ID":"89b3f083-c374-4bbc-b3f2-dda2a50977e4","Type":"ContainerStarted","Data":"de2f6455d101155c253331d8a518dc5460550196538dea86cdea5d95d5da3e5f"} Sep 17 18:04:35.738798 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:35.738684 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" event={"ID":"89b3f083-c374-4bbc-b3f2-dda2a50977e4","Type":"ContainerStarted","Data":"2202aaee7c71b6e63c17824dc0d4735be9cf11de6d3ac2fc3f7ac87286ae178f"} Sep 17 18:04:35.758191 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:35.758147 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5xfxv" podStartSLOduration=33.368354033 podStartE2EDuration="35.758134482s" totalImagesPullingTime="2.389780449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:04:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:35.75789159 +0000 UTC m=+86.821835548" watchObservedRunningTime="2026-09-17 18:04:35.758134482 +0000 UTC m=+86.822078451" Sep 17 18:04:35.781758 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:35.781687 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-7lhd6" podStartSLOduration=33.495877642 podStartE2EDuration="35.781671003s" totalImagesPullingTime="2.285793361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:35.78100464 +0000 UTC m=+86.844948597" watchObservedRunningTime="2026-09-17 18:04:35.781671003 +0000 UTC m=+86.845614962" Sep 17 18:04:35.994408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:35.994342 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7"] Sep 17 18:04:35.999707 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:35.999687 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.002294 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.002276 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 18:04:36.003182 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.003158 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 18:04:36.003381 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.003367 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 18:04:36.004255 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.004242 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 17 18:04:36.027145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.027120 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7"] Sep 17 18:04:36.071856 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.071825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7869c6da-9634-412d-8006-587ada19cc39-tmp\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.072003 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.071908 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7869c6da-9634-412d-8006-587ada19cc39-klusterlet-config\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.072003 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.071946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lrq\" (UniqueName: \"kubernetes.io/projected/7869c6da-9634-412d-8006-587ada19cc39-kube-api-access-65lrq\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.147202 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.147170 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-kfj8l"] Sep 17 18:04:36.150934 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.150919 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-kfj8l" Sep 17 18:04:36.155213 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.155192 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 18:04:36.155819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.155804 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-sgxks\"" Sep 17 18:04:36.155903 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.155823 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 18:04:36.172615 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.172589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7869c6da-9634-412d-8006-587ada19cc39-tmp\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.172746 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.172653 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7869c6da-9634-412d-8006-587ada19cc39-klusterlet-config\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.172746 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.172698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lrq\" (UniqueName: \"kubernetes.io/projected/7869c6da-9634-412d-8006-587ada19cc39-kube-api-access-65lrq\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.172857 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.172796 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-kfj8l"] Sep 17 18:04:36.173452 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.173432 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7869c6da-9634-412d-8006-587ada19cc39-tmp\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.175621 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.175604 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7869c6da-9634-412d-8006-587ada19cc39-klusterlet-config\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.207122 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.207096 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lrq\" (UniqueName: \"kubernetes.io/projected/7869c6da-9634-412d-8006-587ada19cc39-kube-api-access-65lrq\") pod \"klusterlet-addon-workmgr-665b9c664-pgnp7\" (UID: \"7869c6da-9634-412d-8006-587ada19cc39\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.242755 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.242720 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6679f8698c-888bv"] Sep 17 18:04:36.273668 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.273602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wpk\" (UniqueName: \"kubernetes.io/projected/14548da0-26ac-4c8e-b0d9-a5da4aa48090-kube-api-access-w9wpk\") pod \"downloads-6594f4c75-kfj8l\" (UID: \"14548da0-26ac-4c8e-b0d9-a5da4aa48090\") " pod="openshift-console/downloads-6594f4c75-kfj8l" Sep 17 18:04:36.308770 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.308741 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:36.374416 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.374389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wpk\" (UniqueName: \"kubernetes.io/projected/14548da0-26ac-4c8e-b0d9-a5da4aa48090-kube-api-access-w9wpk\") pod \"downloads-6594f4c75-kfj8l\" (UID: \"14548da0-26ac-4c8e-b0d9-a5da4aa48090\") " pod="openshift-console/downloads-6594f4c75-kfj8l" Sep 17 18:04:36.404039 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.404013 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wpk\" (UniqueName: \"kubernetes.io/projected/14548da0-26ac-4c8e-b0d9-a5da4aa48090-kube-api-access-w9wpk\") pod \"downloads-6594f4c75-kfj8l\" (UID: \"14548da0-26ac-4c8e-b0d9-a5da4aa48090\") " pod="openshift-console/downloads-6594f4c75-kfj8l" Sep 17 18:04:36.459642 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.459616 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-kfj8l" Sep 17 18:04:36.590277 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.590193 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7"] Sep 17 18:04:36.593609 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:36.593582 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7869c6da_9634_412d_8006_587ada19cc39.slice/crio-c7215dcad5a6c1a8d9bdfb1e3c7368c667534e91c7210ea319a694dc7cd470d2 WatchSource:0}: Error finding container c7215dcad5a6c1a8d9bdfb1e3c7368c667534e91c7210ea319a694dc7cd470d2: Status 404 returned error can't find the container with id c7215dcad5a6c1a8d9bdfb1e3c7368c667534e91c7210ea319a694dc7cd470d2 Sep 17 18:04:36.743653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.743617 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" event={"ID":"7869c6da-9634-412d-8006-587ada19cc39","Type":"ContainerStarted","Data":"c7215dcad5a6c1a8d9bdfb1e3c7368c667534e91c7210ea319a694dc7cd470d2"} Sep 17 18:04:36.747754 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.747731 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-kfj8l"] Sep 17 18:04:36.749806 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:36.749785 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14548da0_26ac_4c8e_b0d9_a5da4aa48090.slice/crio-da02982b604a362276a1a5e1c9bae1f62ae9fcbf19ac4d3c68ca8da898f43e97 WatchSource:0}: Error finding container da02982b604a362276a1a5e1c9bae1f62ae9fcbf19ac4d3c68ca8da898f43e97: Status 404 returned error can't find the container with id da02982b604a362276a1a5e1c9bae1f62ae9fcbf19ac4d3c68ca8da898f43e97 Sep 17 18:04:36.757589 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.757567 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6986fccc7b-pkstt"] Sep 17 18:04:36.762530 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.762514 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.765177 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.765157 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 18:04:36.765280 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.765209 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 18:04:36.765977 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.765945 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ppsr6\"" Sep 17 18:04:36.765977 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.765957 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 18:04:36.766150 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.766039 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 18:04:36.766329 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.766314 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 18:04:36.774321 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.774300 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6986fccc7b-pkstt"] Sep 17 18:04:36.878286 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.878201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwns\" (UniqueName: \"kubernetes.io/projected/04b7d860-8775-49a9-8849-a3c00cacf03f-kube-api-access-pgwns\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.878286 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.878265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-serving-cert\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.878519 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.878500 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-oauth-serving-cert\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.878579 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.878543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-console-config\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.878663 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.878649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-oauth-config\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.878721 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.878686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-service-ca\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.979761 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.979730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-oauth-config\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.979761 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.979763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-service-ca\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.979953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.979780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwns\" (UniqueName: \"kubernetes.io/projected/04b7d860-8775-49a9-8849-a3c00cacf03f-kube-api-access-pgwns\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.979953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.979798 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-serving-cert\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.979953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.979857 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-oauth-serving-cert\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.979953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.979904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-console-config\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.980562 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.980542 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-oauth-serving-cert\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.980664 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.980586 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-service-ca\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.980664 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.980621 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-console-config\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.982185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.982164 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-serving-cert\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:36.982261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:36.982240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-oauth-config\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:37.002727 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:37.002707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwns\" (UniqueName: \"kubernetes.io/projected/04b7d860-8775-49a9-8849-a3c00cacf03f-kube-api-access-pgwns\") pod \"console-6986fccc7b-pkstt\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:37.071963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:37.071943 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:37.349976 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:37.349945 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6986fccc7b-pkstt"] Sep 17 18:04:37.352921 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:37.352896 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b7d860_8775_49a9_8849_a3c00cacf03f.slice/crio-d3f3898ffd3b4c104a860b1ce04abe911c680052a2324d372e1d654e9363fb78 WatchSource:0}: Error finding container d3f3898ffd3b4c104a860b1ce04abe911c680052a2324d372e1d654e9363fb78: Status 404 returned error can't find the container with id d3f3898ffd3b4c104a860b1ce04abe911c680052a2324d372e1d654e9363fb78 Sep 17 18:04:37.749695 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:37.749657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6986fccc7b-pkstt" event={"ID":"04b7d860-8775-49a9-8849-a3c00cacf03f","Type":"ContainerStarted","Data":"d3f3898ffd3b4c104a860b1ce04abe911c680052a2324d372e1d654e9363fb78"} Sep 17 18:04:37.751597 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:37.751564 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-kfj8l" event={"ID":"14548da0-26ac-4c8e-b0d9-a5da4aa48090","Type":"ContainerStarted","Data":"da02982b604a362276a1a5e1c9bae1f62ae9fcbf19ac4d3c68ca8da898f43e97"} Sep 17 18:04:44.789706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:44.789663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" event={"ID":"7869c6da-9634-412d-8006-587ada19cc39","Type":"ContainerStarted","Data":"71e9515aafe004504c14be51c59426b3bd47faeb036d5c4bc7e4424b1e4945b8"} Sep 17 18:04:44.790233 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:44.789835 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:44.791478 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:44.791453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6986fccc7b-pkstt" event={"ID":"04b7d860-8775-49a9-8849-a3c00cacf03f","Type":"ContainerStarted","Data":"b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c"} Sep 17 18:04:44.791705 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:44.791686 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" Sep 17 18:04:44.854151 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:44.854104 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-665b9c664-pgnp7" podStartSLOduration=2.093021425 podStartE2EDuration="9.854089272s" totalImagesPullingTime="7.761067847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:44.853731311 +0000 UTC m=+95.917675270" watchObservedRunningTime="2026-09-17 18:04:44.854089272 +0000 UTC m=+95.918033274" Sep 17 18:04:44.917126 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:44.917079 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6986fccc7b-pkstt" podStartSLOduration=1.867696441 podStartE2EDuration="8.917064717s" totalImagesPullingTime="7.049368276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:44.916932953 +0000 UTC m=+95.980876916" watchObservedRunningTime="2026-09-17 18:04:44.917064717 +0000 UTC m=+95.981008675" Sep 17 18:04:45.137371 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.137283 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-x8q6b"] Sep 17 18:04:45.139520 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.139501 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qh5nh"] Sep 17 18:04:45.139671 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.139649 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.142204 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.142182 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.143397 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.143358 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-27csq\"" Sep 17 18:04:45.143397 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.143376 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 18:04:45.143569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.143407 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 18:04:45.143683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.143659 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 18:04:45.143986 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.143791 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 18:04:45.144830 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.144808 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 17 18:04:45.144830 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.144826 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 17 18:04:45.145004 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.144925 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-zb8fk\"" Sep 17 18:04:45.145583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.145565 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 17 18:04:45.158159 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.158138 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qh5nh"] Sep 17 18:04:45.259160 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-accelerators-collector-config\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.259326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.259326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6jhv\" (UniqueName: \"kubernetes.io/projected/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-api-access-t6jhv\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.259326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259242 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4c07d5a-7b42-457b-b40d-5325d57e9500-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.259326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259264 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.259326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.259528 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259376 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d4c07d5a-7b42-457b-b40d-5325d57e9500-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.259528 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259402 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-sys\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.259528 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00fbb0f7-be24-4245-bac1-f346974d324e-metrics-client-ca\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.259528 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-textfile\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.259528 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.259528 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-wtmp\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.259791 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhmjt\" (UniqueName: \"kubernetes.io/projected/00fbb0f7-be24-4245-bac1-f346974d324e-kube-api-access-nhmjt\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.259791 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259592 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-tls\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.259791 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.259649 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-root\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.360369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-accelerators-collector-config\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.360544 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360380 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.360544 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jhv\" (UniqueName: \"kubernetes.io/projected/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-api-access-t6jhv\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.360544 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360449 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4c07d5a-7b42-457b-b40d-5325d57e9500-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.360544 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.360544 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360504 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360572 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d4c07d5a-7b42-457b-b40d-5325d57e9500-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-sys\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360642 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-sys\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00fbb0f7-be24-4245-bac1-f346974d324e-metrics-client-ca\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-textfile\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360879 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360929 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-wtmp\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360953 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhmjt\" (UniqueName: \"kubernetes.io/projected/00fbb0f7-be24-4245-bac1-f346974d324e-kube-api-access-nhmjt\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.360989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-tls\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-root\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361067 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-accelerators-collector-config\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-root\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361228 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-textfile\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.361259 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:45.361258 2582 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 17 18:04:45.362042 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:04:45.361325 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-tls podName:d4c07d5a-7b42-457b-b40d-5325d57e9500 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:45.861307544 +0000 UTC m=+96.925251494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-qh5nh" (UID: "d4c07d5a-7b42-457b-b40d-5325d57e9500") : secret "kube-state-metrics-tls" not found Sep 17 18:04:45.362042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361348 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.362042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4c07d5a-7b42-457b-b40d-5325d57e9500-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.362042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-wtmp\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.362042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361545 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d4c07d5a-7b42-457b-b40d-5325d57e9500-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.362042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.361548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00fbb0f7-be24-4245-bac1-f346974d324e-metrics-client-ca\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.363649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.363619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.363761 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.363679 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.363930 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.363912 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/00fbb0f7-be24-4245-bac1-f346974d324e-node-exporter-tls\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.373662 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.373639 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhmjt\" (UniqueName: \"kubernetes.io/projected/00fbb0f7-be24-4245-bac1-f346974d324e-kube-api-access-nhmjt\") pod \"node-exporter-x8q6b\" (UID: \"00fbb0f7-be24-4245-bac1-f346974d324e\") " pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.374893 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.374856 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6jhv\" (UniqueName: \"kubernetes.io/projected/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-api-access-t6jhv\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.452062 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.451977 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-x8q6b" Sep 17 18:04:45.464751 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:45.464726 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fbb0f7_be24_4245_bac1_f346974d324e.slice/crio-c102fd60888ed5100a7757fa345a63c027241ca07565c0cdf3a8fa7fc8fe5983 WatchSource:0}: Error finding container c102fd60888ed5100a7757fa345a63c027241ca07565c0cdf3a8fa7fc8fe5983: Status 404 returned error can't find the container with id c102fd60888ed5100a7757fa345a63c027241ca07565c0cdf3a8fa7fc8fe5983 Sep 17 18:04:45.797122 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.797031 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x8q6b" event={"ID":"00fbb0f7-be24-4245-bac1-f346974d324e","Type":"ContainerStarted","Data":"c102fd60888ed5100a7757fa345a63c027241ca07565c0cdf3a8fa7fc8fe5983"} Sep 17 18:04:45.864943 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.864908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:45.867595 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:45.867568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07d5a-7b42-457b-b40d-5325d57e9500-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-qh5nh\" (UID: \"d4c07d5a-7b42-457b-b40d-5325d57e9500\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:46.059723 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.059645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" Sep 17 18:04:46.067280 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.065542 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:04:46.069569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.069549 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.073621 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.073595 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 18:04:46.073836 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.073820 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 18:04:46.074512 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.074479 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 18:04:46.076019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.074970 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 18:04:46.076019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.075205 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 18:04:46.076019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.075319 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 18:04:46.076019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.075424 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 18:04:46.076019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.075607 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 18:04:46.076019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.075787 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 18:04:46.076019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.075964 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rm44r\"" Sep 17 18:04:46.119158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.119072 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:04:46.168063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.167911 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.167957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.167976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-web-config\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.167997 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168346 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168346 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168161 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168346 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168207 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168346 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168346 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168346 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168305 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flw9h\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-kube-api-access-flw9h\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.168527 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.168352 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-config-out\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.251816 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.251777 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:04:46.269043 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269196 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269196 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269196 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269196 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269472 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flw9h\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-kube-api-access-flw9h\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269515 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-config-out\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269553 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269767 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.269767 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.269624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-web-config\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.271364 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.270395 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.271364 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.270785 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.271364 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.271141 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.272646 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.272619 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.272790 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.272722 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.273143 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.273106 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-web-config\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.273768 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.273652 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-config-volume\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.274982 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.274938 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-config-out\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.275129 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.275109 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.275206 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.275187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.275484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.275458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.276812 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.276771 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.286706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.286680 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flw9h\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-kube-api-access-flw9h\") pod \"alertmanager-main-0\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.365100 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.365077 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-qh5nh"] Sep 17 18:04:46.399264 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.399230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:04:46.559665 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:46.559629 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c07d5a_7b42_457b_b40d_5325d57e9500.slice/crio-24933fdeb40bd3d446e65d15127955a911ba87c5ad9dfd9c0ef146bb9bc6ddbf WatchSource:0}: Error finding container 24933fdeb40bd3d446e65d15127955a911ba87c5ad9dfd9c0ef146bb9bc6ddbf: Status 404 returned error can't find the container with id 24933fdeb40bd3d446e65d15127955a911ba87c5ad9dfd9c0ef146bb9bc6ddbf Sep 17 18:04:46.802765 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.802731 2582 generic.go:356] "Generic (PLEG): container finished" podID="00fbb0f7-be24-4245-bac1-f346974d324e" containerID="5a5da793e601790424c51ab4a12c06c92b68fa2400b8ba7974cf4144569df0ca" exitCode=0 Sep 17 18:04:46.803191 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.802790 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x8q6b" event={"ID":"00fbb0f7-be24-4245-bac1-f346974d324e","Type":"ContainerDied","Data":"5a5da793e601790424c51ab4a12c06c92b68fa2400b8ba7974cf4144569df0ca"} Sep 17 18:04:46.804552 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.804521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" event={"ID":"d4c07d5a-7b42-457b-b40d-5325d57e9500","Type":"ContainerStarted","Data":"24933fdeb40bd3d446e65d15127955a911ba87c5ad9dfd9c0ef146bb9bc6ddbf"} Sep 17 18:04:46.889997 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:46.889944 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:04:46.894631 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:46.894588 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7a80d4_2c22_49ad_858b_db575a586343.slice/crio-9904ac2f0944c2bb2f42b55f79f692bb2288ae8df2c89a9e4ea53cb92aed3705 WatchSource:0}: Error finding container 9904ac2f0944c2bb2f42b55f79f692bb2288ae8df2c89a9e4ea53cb92aed3705: Status 404 returned error can't find the container with id 9904ac2f0944c2bb2f42b55f79f692bb2288ae8df2c89a9e4ea53cb92aed3705 Sep 17 18:04:47.072950 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:47.072916 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:47.073121 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:47.072988 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:47.079350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:47.079322 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:47.814147 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:47.813941 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerStarted","Data":"9904ac2f0944c2bb2f42b55f79f692bb2288ae8df2c89a9e4ea53cb92aed3705"} Sep 17 18:04:47.819116 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:47.819085 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x8q6b" event={"ID":"00fbb0f7-be24-4245-bac1-f346974d324e","Type":"ContainerStarted","Data":"d5c18736a210b0c8c54fe0b68e9e1e48fecb5cf8a8c49739a659acf35214fc5a"} Sep 17 18:04:47.819259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:47.819146 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-x8q6b" event={"ID":"00fbb0f7-be24-4245-bac1-f346974d324e","Type":"ContainerStarted","Data":"956dcfaa0226184fdd7e54cb45e1d4da7665eff1d984d09b517a45ce1caa34b7"} Sep 17 18:04:47.824798 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:47.824773 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:04:47.860752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:47.860027 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-x8q6b" podStartSLOduration=1.662767195 podStartE2EDuration="2.860006203s" totalImagesPullingTime="1.122554088s" totalInitContainerRuntime="74.68492ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:47.858531932 +0000 UTC m=+98.922475892" watchObservedRunningTime="2026-09-17 18:04:47.860006203 +0000 UTC m=+98.923950164" Sep 17 18:04:49.814518 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.814479 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67bd5b5579-kmgxt"] Sep 17 18:04:49.818027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.818005 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:49.826834 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.826811 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 17 18:04:49.826834 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.826826 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 18:04:49.827033 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.826892 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mgl52\"" Sep 17 18:04:49.827162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.827142 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 17 18:04:49.827232 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.827220 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bkk5glufcvj31\"" Sep 17 18:04:49.830385 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.830368 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 17 18:04:49.846158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:49.846133 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67bd5b5579-kmgxt"] Sep 17 18:04:50.007340 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.007294 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3476733-5447-427c-8443-06f4319de249-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.007537 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.007383 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrwk\" (UniqueName: \"kubernetes.io/projected/c3476733-5447-427c-8443-06f4319de249-kube-api-access-qsrwk\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.007537 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.007453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3476733-5447-427c-8443-06f4319de249-metrics-server-audit-profiles\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.007537 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.007486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-client-ca-bundle\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.007537 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.007523 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-secret-metrics-server-tls\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.007737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.007579 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3476733-5447-427c-8443-06f4319de249-audit-log\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.007737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.007635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-secret-metrics-server-client-certs\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.108667 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.108579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-secret-metrics-server-tls\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.108667 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.108628 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3476733-5447-427c-8443-06f4319de249-audit-log\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.108667 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.108665 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-secret-metrics-server-client-certs\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.108966 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.108710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3476733-5447-427c-8443-06f4319de249-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.108966 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.108764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrwk\" (UniqueName: \"kubernetes.io/projected/c3476733-5447-427c-8443-06f4319de249-kube-api-access-qsrwk\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.108966 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.108829 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3476733-5447-427c-8443-06f4319de249-metrics-server-audit-profiles\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.109128 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.108970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-client-ca-bundle\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.109184 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.109162 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c3476733-5447-427c-8443-06f4319de249-audit-log\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.109887 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.109837 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3476733-5447-427c-8443-06f4319de249-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.110174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.110137 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c3476733-5447-427c-8443-06f4319de249-metrics-server-audit-profiles\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.111961 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.111794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-secret-metrics-server-tls\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.111961 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.111903 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-client-ca-bundle\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.111961 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.111905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c3476733-5447-427c-8443-06f4319de249-secret-metrics-server-client-certs\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.125493 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.125471 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrwk\" (UniqueName: \"kubernetes.io/projected/c3476733-5447-427c-8443-06f4319de249-kube-api-access-qsrwk\") pod \"metrics-server-67bd5b5579-kmgxt\" (UID: \"c3476733-5447-427c-8443-06f4319de249\") " pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:50.128330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:50.128253 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:04:51.243774 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.243741 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:04:51.251266 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.251236 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.256147 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.255994 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 18:04:51.256147 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.256070 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 18:04:51.256147 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.256076 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 18:04:51.256495 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.256473 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mmgbt\"" Sep 17 18:04:51.256621 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.256529 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 18:04:51.256621 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.256558 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 18:04:51.256757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.256703 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 18:04:51.256858 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.256835 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 18:04:51.257044 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.257028 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1vi0cqopbgbmr\"" Sep 17 18:04:51.257282 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.257264 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 18:04:51.257680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.257663 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 18:04:51.258322 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.258163 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 18:04:51.258657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.258631 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 18:04:51.258831 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.258809 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 18:04:51.262850 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.262828 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 18:04:51.268232 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.268210 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 18:04:51.271192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.271167 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:04:51.419768 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.419729 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.419768 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.419771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420041 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.419807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420041 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.419922 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420041 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.419950 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420041 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.419978 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420041 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420026 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420276 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420088 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420276 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420276 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420157 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420276 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420276 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420246 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420503 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420291 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420503 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420320 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420503 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420364 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420503 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420503 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzcbn\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-kube-api-access-qzcbn\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420503 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420466 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420735 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.420735 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.420550 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521159 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521159 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521400 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521283 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521400 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521400 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521400 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521483 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521512 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzcbn\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-kube-api-access-qzcbn\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521525 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521540 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521912 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521601 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521912 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521912 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521912 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521912 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.521912 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.522245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521946 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.522245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.521974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.522245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.522001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.522395 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.522303 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.522820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.522790 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.524359 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.523339 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.524359 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.524105 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.525482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.524792 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.525482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.525191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.525482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.525197 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.525482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.525216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.525739 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.525512 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.526205 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.526180 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.526296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.526203 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.526709 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.526647 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.527703 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.527682 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-web-config\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.528125 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.528081 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config-out\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.528700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.528380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.528700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.528584 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.528700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.528628 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.528881 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.528814 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.542395 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.542371 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzcbn\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-kube-api-access-qzcbn\") pod \"prometheus-k8s-0\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.565135 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.565109 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:04:51.823946 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.823846 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:04:51.826359 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:51.826325 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aeee9793-7091-4047-b99c-871e17adafa0-metrics-tls\") pod \"dns-default-42kg8\" (UID: \"aeee9793-7091-4047-b99c-871e17adafa0\") " pod="openshift-dns/dns-default-42kg8" Sep 17 18:04:52.050004 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:52.049979 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dfk9\"" Sep 17 18:04:52.058281 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:52.058249 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-42kg8" Sep 17 18:04:52.677656 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:52.677621 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wrnpc" Sep 17 18:04:56.086230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.086202 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-42kg8"] Sep 17 18:04:56.090773 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.090687 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67bd5b5579-kmgxt"] Sep 17 18:04:56.093653 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:56.093615 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeee9793_7091_4047_b99c_871e17adafa0.slice/crio-a110682e1f96a42af6d9ff949cf82dd458cc7deb03c7e68fc9a6293cddeb9369 WatchSource:0}: Error finding container a110682e1f96a42af6d9ff949cf82dd458cc7deb03c7e68fc9a6293cddeb9369: Status 404 returned error can't find the container with id a110682e1f96a42af6d9ff949cf82dd458cc7deb03c7e68fc9a6293cddeb9369 Sep 17 18:04:56.118067 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.118043 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:04:56.125832 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:04:56.125768 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d78faaa_2cc2_4511_beb4_34bfd88e098f.slice/crio-385f77bc5dc2926ec10bec751fab49745ab92d0c92b63c58895a6e020fe91aaa WatchSource:0}: Error finding container 385f77bc5dc2926ec10bec751fab49745ab92d0c92b63c58895a6e020fe91aaa: Status 404 returned error can't find the container with id 385f77bc5dc2926ec10bec751fab49745ab92d0c92b63c58895a6e020fe91aaa Sep 17 18:04:56.876142 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.875897 2582 generic.go:356] "Generic (PLEG): container finished" podID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" exitCode=0 Sep 17 18:04:56.876142 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.876023 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerDied","Data":"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647"} Sep 17 18:04:56.876142 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.876091 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerStarted","Data":"385f77bc5dc2926ec10bec751fab49745ab92d0c92b63c58895a6e020fe91aaa"} Sep 17 18:04:56.881438 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.880505 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7a80d4-2c22-49ad-858b-db575a586343" containerID="6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf" exitCode=0 Sep 17 18:04:56.881438 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.881358 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerDied","Data":"6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf"} Sep 17 18:04:56.884631 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.884556 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" event={"ID":"c3476733-5447-427c-8443-06f4319de249","Type":"ContainerStarted","Data":"0cb41244e9122c6a2f2c017e6229b571a7f4d29083a42112b0e381967b01d7fd"} Sep 17 18:04:56.889368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.888648 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-kfj8l" event={"ID":"14548da0-26ac-4c8e-b0d9-a5da4aa48090","Type":"ContainerStarted","Data":"5883dde4846b0090cccf77171e16fe74dde6fa08b74b2dfba9381006c564db56"} Sep 17 18:04:56.889368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.889307 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-kfj8l" Sep 17 18:04:56.892045 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.892025 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42kg8" event={"ID":"aeee9793-7091-4047-b99c-871e17adafa0","Type":"ContainerStarted","Data":"a110682e1f96a42af6d9ff949cf82dd458cc7deb03c7e68fc9a6293cddeb9369"} Sep 17 18:04:56.896019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.895992 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" event={"ID":"d4c07d5a-7b42-457b-b40d-5325d57e9500","Type":"ContainerStarted","Data":"9febbd2fd5b3bd9b6e0491fb845844ed90e9b2ea0eba695e623c42ab84814b50"} Sep 17 18:04:56.896123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.896054 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" event={"ID":"d4c07d5a-7b42-457b-b40d-5325d57e9500","Type":"ContainerStarted","Data":"4c27fd4986974c1e1938aab76ab766ac0f68f75e8f630e46d741158e6a5d66ab"} Sep 17 18:04:56.896123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.896090 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" event={"ID":"d4c07d5a-7b42-457b-b40d-5325d57e9500","Type":"ContainerStarted","Data":"5710d459278f3a510781a5adc3577c4144af403c4f4b66a3c731ac6b4d9386ce"} Sep 17 18:04:56.910940 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:56.910914 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-kfj8l" Sep 17 18:04:57.045118 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:57.042827 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-kfj8l" podStartSLOduration=1.944576982 podStartE2EDuration="21.042801645s" totalImagesPullingTime="19.098224663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:57.014579463 +0000 UTC m=+108.078523422" watchObservedRunningTime="2026-09-17 18:04:57.042801645 +0000 UTC m=+108.106745604" Sep 17 18:04:57.045118 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:04:57.043177 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-qh5nh" podStartSLOduration=2.82186213 podStartE2EDuration="12.043167393s" totalImagesPullingTime="9.221305263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:57.041437739 +0000 UTC m=+108.105381699" watchObservedRunningTime="2026-09-17 18:04:57.043167393 +0000 UTC m=+108.107111346" Sep 17 18:05:01.261130 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:01.261044 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6679f8698c-888bv" podUID="152119e1-0219-4f51-97ba-6e06e41b8487" containerName="registry" containerID="cri-o://42579c28f6825c2632c2bc0276141dec2fb06494a2a82ee58a8735b568d70c1c" gracePeriod=30 Sep 17 18:05:01.927112 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:01.927075 2582 generic.go:356] "Generic (PLEG): container finished" podID="152119e1-0219-4f51-97ba-6e06e41b8487" containerID="42579c28f6825c2632c2bc0276141dec2fb06494a2a82ee58a8735b568d70c1c" exitCode=0 Sep 17 18:05:01.927294 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:01.927205 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6679f8698c-888bv" event={"ID":"152119e1-0219-4f51-97ba-6e06e41b8487","Type":"ContainerDied","Data":"42579c28f6825c2632c2bc0276141dec2fb06494a2a82ee58a8735b568d70c1c"} Sep 17 18:05:02.043479 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.043450 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.129752 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-image-registry-private-configuration\") pod \"152119e1-0219-4f51-97ba-6e06e41b8487\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.129814 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-installation-pull-secrets\") pod \"152119e1-0219-4f51-97ba-6e06e41b8487\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.129851 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152119e1-0219-4f51-97ba-6e06e41b8487-ca-trust-extracted\") pod \"152119e1-0219-4f51-97ba-6e06e41b8487\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.129923 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") pod \"152119e1-0219-4f51-97ba-6e06e41b8487\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.129968 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-bound-sa-token\") pod \"152119e1-0219-4f51-97ba-6e06e41b8487\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.130017 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-trusted-ca\") pod \"152119e1-0219-4f51-97ba-6e06e41b8487\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.130054 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm5xg\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-kube-api-access-bm5xg\") pod \"152119e1-0219-4f51-97ba-6e06e41b8487\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.130102 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-registry-certificates\") pod \"152119e1-0219-4f51-97ba-6e06e41b8487\" (UID: \"152119e1-0219-4f51-97ba-6e06e41b8487\") " Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.130727 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "152119e1-0219-4f51-97ba-6e06e41b8487" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:02.131248 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.130939 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "152119e1-0219-4f51-97ba-6e06e41b8487" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:02.133576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.133520 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "152119e1-0219-4f51-97ba-6e06e41b8487" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:02.138382 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.138325 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "152119e1-0219-4f51-97ba-6e06e41b8487" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:02.139430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.139372 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "152119e1-0219-4f51-97ba-6e06e41b8487" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:02.140283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.140231 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "152119e1-0219-4f51-97ba-6e06e41b8487" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:02.141233 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.140970 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-kube-api-access-bm5xg" (OuterVolumeSpecName: "kube-api-access-bm5xg") pod "152119e1-0219-4f51-97ba-6e06e41b8487" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487"). InnerVolumeSpecName "kube-api-access-bm5xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:02.144437 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.144399 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152119e1-0219-4f51-97ba-6e06e41b8487-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "152119e1-0219-4f51-97ba-6e06e41b8487" (UID: "152119e1-0219-4f51-97ba-6e06e41b8487"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:05:02.231470 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.231433 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-bound-sa-token\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:02.231470 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.231468 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-trusted-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:02.231470 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.231480 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm5xg\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-kube-api-access-bm5xg\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:02.231698 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.231492 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/152119e1-0219-4f51-97ba-6e06e41b8487-registry-certificates\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:02.231698 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.231504 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-image-registry-private-configuration\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:02.231698 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.231516 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/152119e1-0219-4f51-97ba-6e06e41b8487-installation-pull-secrets\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:02.231698 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.231526 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/152119e1-0219-4f51-97ba-6e06e41b8487-ca-trust-extracted\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:02.231698 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.231539 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/152119e1-0219-4f51-97ba-6e06e41b8487-registry-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:02.935668 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.935626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42kg8" event={"ID":"aeee9793-7091-4047-b99c-871e17adafa0","Type":"ContainerStarted","Data":"d5669635e975b97026b6886e2c4534c80245ed78705b4a52b09b6bbdee3d9d0f"} Sep 17 18:05:02.936252 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.935736 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42kg8" event={"ID":"aeee9793-7091-4047-b99c-871e17adafa0","Type":"ContainerStarted","Data":"1c311ed7b4a3db3742b2612c1ce0fb9601be9b4e274d674856819ea952a0cd10"} Sep 17 18:05:02.936252 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.935910 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-42kg8" Sep 17 18:05:02.938978 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.938699 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerStarted","Data":"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8"} Sep 17 18:05:02.938978 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.938757 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerStarted","Data":"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff"} Sep 17 18:05:02.942703 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.942682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerStarted","Data":"2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac"} Sep 17 18:05:02.942817 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.942748 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerStarted","Data":"6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749"} Sep 17 18:05:02.942817 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.942793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerStarted","Data":"0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4"} Sep 17 18:05:02.942943 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.942832 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerStarted","Data":"b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088"} Sep 17 18:05:02.942943 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.942888 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerStarted","Data":"bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5"} Sep 17 18:05:02.944722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.944700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" event={"ID":"c3476733-5447-427c-8443-06f4319de249","Type":"ContainerStarted","Data":"ca5488906410d36fec3e35089fac9adcd90425d41db25b3468f04de329ea0847"} Sep 17 18:05:02.946263 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.946241 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6679f8698c-888bv" event={"ID":"152119e1-0219-4f51-97ba-6e06e41b8487","Type":"ContainerDied","Data":"e1ef7ffa7c6e7566a601562b81fc687cb04b0b463d3f9261b14bce8a2b5e4a5d"} Sep 17 18:05:02.946397 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.946379 2582 scope.go:117] "RemoveContainer" containerID="42579c28f6825c2632c2bc0276141dec2fb06494a2a82ee58a8735b568d70c1c" Sep 17 18:05:02.946517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.946298 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6679f8698c-888bv" Sep 17 18:05:02.965238 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.965188 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-42kg8" podStartSLOduration=70.163524774 podStartE2EDuration="1m15.965172003s" totalImagesPullingTime="5.801647229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:02.963500486 +0000 UTC m=+114.027444445" watchObservedRunningTime="2026-09-17 18:05:02.965172003 +0000 UTC m=+114.029115962" Sep 17 18:05:02.982566 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.982539 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6679f8698c-888bv"] Sep 17 18:05:02.990145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:02.990114 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6679f8698c-888bv"] Sep 17 18:05:03.021102 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:03.021051 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" podStartSLOduration=8.222808597 podStartE2EDuration="14.021035756s" totalImagesPullingTime="5.798227159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:03.01954017 +0000 UTC m=+114.083484129" watchObservedRunningTime="2026-09-17 18:05:03.021035756 +0000 UTC m=+114.084979714" Sep 17 18:05:03.347927 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:03.347313 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="152119e1-0219-4f51-97ba-6e06e41b8487" path="/var/lib/kubelet/pods/152119e1-0219-4f51-97ba-6e06e41b8487/volumes" Sep 17 18:05:05.965270 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:05.965227 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerStarted","Data":"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f"} Sep 17 18:05:05.965594 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:05.965349 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerStarted","Data":"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4"} Sep 17 18:05:05.965594 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:05.965393 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerStarted","Data":"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6"} Sep 17 18:05:05.969069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:05.969046 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerStarted","Data":"f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203"} Sep 17 18:05:06.018909 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:06.018837 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.9527861 podStartE2EDuration="20.018814191s" totalImagesPullingTime="16.048940147s" totalInitContainerRuntime="17.087944ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:06.017129638 +0000 UTC m=+117.081073621" watchObservedRunningTime="2026-09-17 18:05:06.018814191 +0000 UTC m=+117.082758153" Sep 17 18:05:06.977162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:06.977123 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerStarted","Data":"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c"} Sep 17 18:05:07.022395 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:07.022331 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.671301197 podStartE2EDuration="16.022303422s" totalImagesPullingTime="8.326741128s" totalInitContainerRuntime="24.261097ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:07.019637636 +0000 UTC m=+118.083581607" watchObservedRunningTime="2026-09-17 18:05:07.022303422 +0000 UTC m=+118.086247383" Sep 17 18:05:07.210327 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:07.210298 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6986fccc7b-pkstt"] Sep 17 18:05:10.129179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:10.129144 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:05:10.129659 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:10.129210 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:05:11.565255 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:11.565167 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:05:12.955315 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:12.955278 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-42kg8" Sep 17 18:05:20.201021 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:20.200982 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:05:20.203320 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:20.203296 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dd86bb7-9b68-4957-87ba-905e2ffe4948-metrics-certs\") pod \"network-metrics-daemon-ctrs8\" (UID: \"9dd86bb7-9b68-4957-87ba-905e2ffe4948\") " pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:05:20.260116 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:20.260088 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qc24z\"" Sep 17 18:05:20.268419 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:20.268398 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ctrs8" Sep 17 18:05:20.563894 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:20.563701 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ctrs8"] Sep 17 18:05:20.566540 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:05:20.566514 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd86bb7_9b68_4957_87ba_905e2ffe4948.slice/crio-70cce5a986978ab619252d684f6f98c9a1fc9a16aee7dee17386243881c6f790 WatchSource:0}: Error finding container 70cce5a986978ab619252d684f6f98c9a1fc9a16aee7dee17386243881c6f790: Status 404 returned error can't find the container with id 70cce5a986978ab619252d684f6f98c9a1fc9a16aee7dee17386243881c6f790 Sep 17 18:05:21.032164 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:21.032128 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctrs8" event={"ID":"9dd86bb7-9b68-4957-87ba-905e2ffe4948","Type":"ContainerStarted","Data":"70cce5a986978ab619252d684f6f98c9a1fc9a16aee7dee17386243881c6f790"} Sep 17 18:05:22.037931 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:22.037897 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctrs8" event={"ID":"9dd86bb7-9b68-4957-87ba-905e2ffe4948","Type":"ContainerStarted","Data":"7ec7c35d956b3d6fd2e2ebbf2315e9ac7c9d082f490e48583c92f00cf31e1406"} Sep 17 18:05:23.043922 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:23.043885 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ctrs8" event={"ID":"9dd86bb7-9b68-4957-87ba-905e2ffe4948","Type":"ContainerStarted","Data":"b50370690e3e71ac3bca1a76147863ee6eabac50ef7e346fd04ac67d4845f065"} Sep 17 18:05:23.075601 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:23.075546 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ctrs8" podStartSLOduration=130.772144012 podStartE2EDuration="2m12.07553023s" totalImagesPullingTime="1.303386218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:05:23.074191268 +0000 UTC m=+134.138135218" watchObservedRunningTime="2026-09-17 18:05:23.07553023 +0000 UTC m=+134.139474188" Sep 17 18:05:30.135212 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:30.135181 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:05:30.139394 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:30.139371 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67bd5b5579-kmgxt" Sep 17 18:05:32.229763 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.229717 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6986fccc7b-pkstt" podUID="04b7d860-8775-49a9-8849-a3c00cacf03f" containerName="console" containerID="cri-o://b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c" gracePeriod=15 Sep 17 18:05:32.519887 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.519833 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:05:32.610438 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610406 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgwns\" (UniqueName: \"kubernetes.io/projected/04b7d860-8775-49a9-8849-a3c00cacf03f-kube-api-access-pgwns\") pod \"04b7d860-8775-49a9-8849-a3c00cacf03f\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " Sep 17 18:05:32.610438 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610440 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-oauth-serving-cert\") pod \"04b7d860-8775-49a9-8849-a3c00cacf03f\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " Sep 17 18:05:32.610649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610456 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-console-config\") pod \"04b7d860-8775-49a9-8849-a3c00cacf03f\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " Sep 17 18:05:32.610649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610502 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-service-ca\") pod \"04b7d860-8775-49a9-8849-a3c00cacf03f\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " Sep 17 18:05:32.610649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610542 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-serving-cert\") pod \"04b7d860-8775-49a9-8849-a3c00cacf03f\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " Sep 17 18:05:32.610649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610606 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-oauth-config\") pod \"04b7d860-8775-49a9-8849-a3c00cacf03f\" (UID: \"04b7d860-8775-49a9-8849-a3c00cacf03f\") " Sep 17 18:05:32.610958 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610920 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "04b7d860-8775-49a9-8849-a3c00cacf03f" (UID: "04b7d860-8775-49a9-8849-a3c00cacf03f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:32.610958 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610929 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-console-config" (OuterVolumeSpecName: "console-config") pod "04b7d860-8775-49a9-8849-a3c00cacf03f" (UID: "04b7d860-8775-49a9-8849-a3c00cacf03f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:32.611133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.610962 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-service-ca" (OuterVolumeSpecName: "service-ca") pod "04b7d860-8775-49a9-8849-a3c00cacf03f" (UID: "04b7d860-8775-49a9-8849-a3c00cacf03f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:05:32.612847 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.612820 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "04b7d860-8775-49a9-8849-a3c00cacf03f" (UID: "04b7d860-8775-49a9-8849-a3c00cacf03f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:32.612965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.612911 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b7d860-8775-49a9-8849-a3c00cacf03f-kube-api-access-pgwns" (OuterVolumeSpecName: "kube-api-access-pgwns") pod "04b7d860-8775-49a9-8849-a3c00cacf03f" (UID: "04b7d860-8775-49a9-8849-a3c00cacf03f"). InnerVolumeSpecName "kube-api-access-pgwns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:05:32.612965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.612949 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "04b7d860-8775-49a9-8849-a3c00cacf03f" (UID: "04b7d860-8775-49a9-8849-a3c00cacf03f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:05:32.711901 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.711858 2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-serving-cert\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:32.711901 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.711901 2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/04b7d860-8775-49a9-8849-a3c00cacf03f-console-oauth-config\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:32.712096 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.711914 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgwns\" (UniqueName: \"kubernetes.io/projected/04b7d860-8775-49a9-8849-a3c00cacf03f-kube-api-access-pgwns\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:32.712096 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.711927 2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-oauth-serving-cert\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:32.712096 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.711941 2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-console-config\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:32.712096 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:32.711953 2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/04b7d860-8775-49a9-8849-a3c00cacf03f-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:05:33.086473 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.086433 2582 generic.go:356] "Generic (PLEG): container finished" podID="04b7d860-8775-49a9-8849-a3c00cacf03f" containerID="b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c" exitCode=0 Sep 17 18:05:33.086666 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.086484 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6986fccc7b-pkstt" Sep 17 18:05:33.086666 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.086504 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6986fccc7b-pkstt" event={"ID":"04b7d860-8775-49a9-8849-a3c00cacf03f","Type":"ContainerDied","Data":"b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c"} Sep 17 18:05:33.086666 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.086577 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6986fccc7b-pkstt" event={"ID":"04b7d860-8775-49a9-8849-a3c00cacf03f","Type":"ContainerDied","Data":"d3f3898ffd3b4c104a860b1ce04abe911c680052a2324d372e1d654e9363fb78"} Sep 17 18:05:33.086666 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.086600 2582 scope.go:117] "RemoveContainer" containerID="b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c" Sep 17 18:05:33.095432 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.095415 2582 scope.go:117] "RemoveContainer" containerID="b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c" Sep 17 18:05:33.095721 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:05:33.095699 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c\": container with ID starting with b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c not found: ID does not exist" containerID="b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c" Sep 17 18:05:33.095793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.095729 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c"} err="failed to get container status \"b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c\": rpc error: code = NotFound desc = could not find container \"b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c\": container with ID starting with b97c8fc88c7d4b9c6bfb61a250cc5c7dfa1f3f08a5fd577211d76f2cd2bee91c not found: ID does not exist" Sep 17 18:05:33.111228 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.111201 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6986fccc7b-pkstt"] Sep 17 18:05:33.115834 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.115812 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6986fccc7b-pkstt"] Sep 17 18:05:33.344174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:33.344100 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04b7d860-8775-49a9-8849-a3c00cacf03f" path="/var/lib/kubelet/pods/04b7d860-8775-49a9-8849-a3c00cacf03f/volumes" Sep 17 18:05:41.123048 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:41.123008 2582 generic.go:356] "Generic (PLEG): container finished" podID="661c1262-8856-4aa9-be54-f3a2594cee26" containerID="f425edb87baac03f7833e3acb7e1496183d5165bac11e52fb95bf271835b4786" exitCode=0 Sep 17 18:05:41.123474 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:41.123061 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" event={"ID":"661c1262-8856-4aa9-be54-f3a2594cee26","Type":"ContainerDied","Data":"f425edb87baac03f7833e3acb7e1496183d5165bac11e52fb95bf271835b4786"} Sep 17 18:05:41.123474 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:41.123332 2582 scope.go:117] "RemoveContainer" containerID="f425edb87baac03f7833e3acb7e1496183d5165bac11e52fb95bf271835b4786" Sep 17 18:05:42.128673 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:42.128635 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-frch6" event={"ID":"661c1262-8856-4aa9-be54-f3a2594cee26","Type":"ContainerStarted","Data":"3f80a032cad9ce150e94dd7106887a62c8b19de2af72eadfbe1dd01432d00421"} Sep 17 18:05:51.565576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:51.565527 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:05:51.584803 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:51.584775 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:05:52.187163 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:05:52.187135 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:05.504952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:05.504913 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:06:05.505517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:05.505195 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="alertmanager" containerID="cri-o://bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5" gracePeriod=120 Sep 17 18:06:05.505517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:05.505227 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy" containerID="cri-o://6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749" gracePeriod=120 Sep 17 18:06:05.505517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:05.505262 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy-web" containerID="cri-o://0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4" gracePeriod=120 Sep 17 18:06:05.505517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:05.505272 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy-metric" containerID="cri-o://2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac" gracePeriod=120 Sep 17 18:06:05.505517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:05.505291 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="config-reloader" containerID="cri-o://b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088" gracePeriod=120 Sep 17 18:06:05.505517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:05.505234 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="prom-label-proxy" containerID="cri-o://f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203" gracePeriod=120 Sep 17 18:06:06.227269 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227237 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7a80d4-2c22-49ad-858b-db575a586343" containerID="f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203" exitCode=0 Sep 17 18:06:06.227269 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227263 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7a80d4-2c22-49ad-858b-db575a586343" containerID="2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac" exitCode=0 Sep 17 18:06:06.227269 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227271 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7a80d4-2c22-49ad-858b-db575a586343" containerID="6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749" exitCode=0 Sep 17 18:06:06.227269 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227277 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7a80d4-2c22-49ad-858b-db575a586343" containerID="b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088" exitCode=0 Sep 17 18:06:06.227269 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227283 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7a80d4-2c22-49ad-858b-db575a586343" containerID="bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5" exitCode=0 Sep 17 18:06:06.227609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerDied","Data":"f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203"} Sep 17 18:06:06.227609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerDied","Data":"2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac"} Sep 17 18:06:06.227609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227411 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerDied","Data":"6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749"} Sep 17 18:06:06.227609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227435 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerDied","Data":"b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088"} Sep 17 18:06:06.227609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.227459 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerDied","Data":"bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5"} Sep 17 18:06:06.769482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.769455 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:06.906954 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.906929 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-cluster-tls-config\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.906964 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-web-config\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.906993 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-main-tls\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907032 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-tls-assets\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907068 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-web\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907105 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-config-volume\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907374 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907238 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-metrics-client-ca\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907374 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907277 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-config-out\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907374 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907309 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-trusted-ca-bundle\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907374 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907342 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-main-db\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907374 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907369 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flw9h\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-kube-api-access-flw9h\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907604 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907458 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907604 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907498 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy\") pod \"6f7a80d4-2c22-49ad-858b-db575a586343\" (UID: \"6f7a80d4-2c22-49ad-858b-db575a586343\") " Sep 17 18:06:06.907604 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907570 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:06.907903 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.907790 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-metrics-client-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:06.908031 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.908000 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:06:06.908612 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.908554 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:06.909716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.909677 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:06:06.910498 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.910459 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:06.910775 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.910726 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:06.910951 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.910932 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-config-out" (OuterVolumeSpecName: "config-out") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:06:06.911056 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.911033 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-config-volume" (OuterVolumeSpecName: "config-volume") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:06.911145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.911119 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:06.911394 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.911369 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-kube-api-access-flw9h" (OuterVolumeSpecName: "kube-api-access-flw9h") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "kube-api-access-flw9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:06:06.911635 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.911617 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:06.914325 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.914303 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:06.920970 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:06.920949 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-web-config" (OuterVolumeSpecName: "web-config") pod "6f7a80d4-2c22-49ad-858b-db575a586343" (UID: "6f7a80d4-2c22-49ad-858b-db575a586343"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:07.009197 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009170 2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-main-db\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009197 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009193 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flw9h\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-kube-api-access-flw9h\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009203 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009214 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009222 2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-cluster-tls-config\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009231 2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-web-config\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009239 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-main-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009247 2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6f7a80d4-2c22-49ad-858b-db575a586343-tls-assets\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009255 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009262 2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6f7a80d4-2c22-49ad-858b-db575a586343-config-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009270 2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6f7a80d4-2c22-49ad-858b-db575a586343-config-out\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.009332 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.009277 2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f7a80d4-2c22-49ad-858b-db575a586343-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:07.235027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.234950 2582 generic.go:356] "Generic (PLEG): container finished" podID="6f7a80d4-2c22-49ad-858b-db575a586343" containerID="0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4" exitCode=0 Sep 17 18:06:07.235027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.235016 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerDied","Data":"0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4"} Sep 17 18:06:07.235201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.235035 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.235201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.235074 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6f7a80d4-2c22-49ad-858b-db575a586343","Type":"ContainerDied","Data":"9904ac2f0944c2bb2f42b55f79f692bb2288ae8df2c89a9e4ea53cb92aed3705"} Sep 17 18:06:07.235201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.235106 2582 scope.go:117] "RemoveContainer" containerID="f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203" Sep 17 18:06:07.243303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.243284 2582 scope.go:117] "RemoveContainer" containerID="2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac" Sep 17 18:06:07.250071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.250053 2582 scope.go:117] "RemoveContainer" containerID="6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749" Sep 17 18:06:07.256405 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.256389 2582 scope.go:117] "RemoveContainer" containerID="0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4" Sep 17 18:06:07.263600 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.263227 2582 scope.go:117] "RemoveContainer" containerID="b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088" Sep 17 18:06:07.264134 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.264112 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:06:07.271262 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.271245 2582 scope.go:117] "RemoveContainer" containerID="bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5" Sep 17 18:06:07.272090 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.272068 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:06:07.277978 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.277960 2582 scope.go:117] "RemoveContainer" containerID="6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf" Sep 17 18:06:07.284585 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.284569 2582 scope.go:117] "RemoveContainer" containerID="f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203" Sep 17 18:06:07.284831 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:07.284812 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203\": container with ID starting with f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203 not found: ID does not exist" containerID="f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203" Sep 17 18:06:07.284917 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.284840 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203"} err="failed to get container status \"f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203\": rpc error: code = NotFound desc = could not find container \"f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203\": container with ID starting with f69549decdafe4c86026bcbd5369b107dab0d6a4e3a9840d7a8482c33ebae203 not found: ID does not exist" Sep 17 18:06:07.284917 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.284858 2582 scope.go:117] "RemoveContainer" containerID="2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac" Sep 17 18:06:07.285139 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:07.285120 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac\": container with ID starting with 2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac not found: ID does not exist" containerID="2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac" Sep 17 18:06:07.285177 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.285148 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac"} err="failed to get container status \"2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac\": rpc error: code = NotFound desc = could not find container \"2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac\": container with ID starting with 2e8a3b20854bff38a0d1bf67dc6a3dc6d8dc7e56b1908dcf0642ce204ee421ac not found: ID does not exist" Sep 17 18:06:07.285177 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.285171 2582 scope.go:117] "RemoveContainer" containerID="6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749" Sep 17 18:06:07.285402 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:07.285387 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749\": container with ID starting with 6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749 not found: ID does not exist" containerID="6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749" Sep 17 18:06:07.285449 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.285408 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749"} err="failed to get container status \"6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749\": rpc error: code = NotFound desc = could not find container \"6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749\": container with ID starting with 6379b5a059a549134aad7bc3f31f560d3f2a5957032ce910f144ae369da61749 not found: ID does not exist" Sep 17 18:06:07.285449 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.285422 2582 scope.go:117] "RemoveContainer" containerID="0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4" Sep 17 18:06:07.285643 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:07.285624 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4\": container with ID starting with 0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4 not found: ID does not exist" containerID="0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4" Sep 17 18:06:07.285682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.285647 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4"} err="failed to get container status \"0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4\": rpc error: code = NotFound desc = could not find container \"0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4\": container with ID starting with 0d2ef3687168c508419d7566a41a590c30f243d2c09e24411c0ff31dee3094a4 not found: ID does not exist" Sep 17 18:06:07.285682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.285659 2582 scope.go:117] "RemoveContainer" containerID="b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088" Sep 17 18:06:07.285894 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:07.285875 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088\": container with ID starting with b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088 not found: ID does not exist" containerID="b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088" Sep 17 18:06:07.285947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.285901 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088"} err="failed to get container status \"b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088\": rpc error: code = NotFound desc = could not find container \"b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088\": container with ID starting with b24ce45957c07600e7efe9280afcbbfd63980f4fb6d23e7674a3fa10320a2088 not found: ID does not exist" Sep 17 18:06:07.285947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.285915 2582 scope.go:117] "RemoveContainer" containerID="bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5" Sep 17 18:06:07.286177 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:07.286162 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5\": container with ID starting with bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5 not found: ID does not exist" containerID="bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5" Sep 17 18:06:07.286222 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.286182 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5"} err="failed to get container status \"bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5\": rpc error: code = NotFound desc = could not find container \"bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5\": container with ID starting with bc00c09b3880dca8a021ab15f171e7bd80ed5a0602bafa48c22822dd1c0d91e5 not found: ID does not exist" Sep 17 18:06:07.286222 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.286195 2582 scope.go:117] "RemoveContainer" containerID="6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf" Sep 17 18:06:07.286406 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:07.286391 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf\": container with ID starting with 6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf not found: ID does not exist" containerID="6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf" Sep 17 18:06:07.286446 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.286409 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf"} err="failed to get container status \"6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf\": rpc error: code = NotFound desc = could not find container \"6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf\": container with ID starting with 6651f82af29b20ea046be5355e4036b63ddcbf544cf70dff9d92dc91d4f776bf not found: ID does not exist" Sep 17 18:06:07.310697 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.310674 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:06:07.311013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.310999 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152119e1-0219-4f51-97ba-6e06e41b8487" containerName="registry" Sep 17 18:06:07.311071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311016 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="152119e1-0219-4f51-97ba-6e06e41b8487" containerName="registry" Sep 17 18:06:07.311071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311030 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy-web" Sep 17 18:06:07.311071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311036 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy-web" Sep 17 18:06:07.311071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311043 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy" Sep 17 18:06:07.311071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311048 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy" Sep 17 18:06:07.311071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311058 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="prom-label-proxy" Sep 17 18:06:07.311071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311064 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="prom-label-proxy" Sep 17 18:06:07.311071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311071 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="config-reloader" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311077 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="config-reloader" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311084 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy-metric" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311090 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy-metric" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311097 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04b7d860-8775-49a9-8849-a3c00cacf03f" containerName="console" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311102 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04b7d860-8775-49a9-8849-a3c00cacf03f" containerName="console" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311112 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="init-config-reloader" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311117 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="init-config-reloader" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311126 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="alertmanager" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311132 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="alertmanager" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311178 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="152119e1-0219-4f51-97ba-6e06e41b8487" containerName="registry" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311186 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04b7d860-8775-49a9-8849-a3c00cacf03f" containerName="console" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311191 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="alertmanager" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311196 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="config-reloader" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311200 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311206 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy-metric" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311211 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="prom-label-proxy" Sep 17 18:06:07.311316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.311216 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" containerName="kube-rbac-proxy-web" Sep 17 18:06:07.317740 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.317718 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.322291 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.322272 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 18:06:07.322458 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.322440 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 18:06:07.322771 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.322729 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 18:06:07.323002 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.322978 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 18:06:07.323303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.323270 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 18:06:07.323637 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.323616 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rm44r\"" Sep 17 18:06:07.323737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.323642 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 18:06:07.323891 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.323858 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 18:06:07.323961 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.323946 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 18:06:07.330871 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.330847 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 18:06:07.334857 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.334834 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:06:07.344185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.344157 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f7a80d4-2c22-49ad-858b-db575a586343" path="/var/lib/kubelet/pods/6f7a80d4-2c22-49ad-858b-db575a586343/volumes" Sep 17 18:06:07.414238 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1966b53-b777-4ad3-831e-0598bb3938e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414361 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414245 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1966b53-b777-4ad3-831e-0598bb3938e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414361 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1966b53-b777-4ad3-831e-0598bb3938e9-config-out\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414361 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414278 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414473 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1966b53-b777-4ad3-831e-0598bb3938e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414473 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414393 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414473 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414416 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1966b53-b777-4ad3-831e-0598bb3938e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414473 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414441 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-web-config\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414473 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414647 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414487 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414647 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414647 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414518 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.414647 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.414538 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4c5d\" (UniqueName: \"kubernetes.io/projected/d1966b53-b777-4ad3-831e-0598bb3938e9-kube-api-access-l4c5d\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.515899 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.515814 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1966b53-b777-4ad3-831e-0598bb3938e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.515899 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.515851 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-web-config\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.515899 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.515899 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.515920 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.515937 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.515962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.515993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4c5d\" (UniqueName: \"kubernetes.io/projected/d1966b53-b777-4ad3-831e-0598bb3938e9-kube-api-access-l4c5d\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.516050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1966b53-b777-4ad3-831e-0598bb3938e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.516082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1966b53-b777-4ad3-831e-0598bb3938e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.516105 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1966b53-b777-4ad3-831e-0598bb3938e9-config-out\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.516132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.516167 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1966b53-b777-4ad3-831e-0598bb3938e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.516195 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.516498 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d1966b53-b777-4ad3-831e-0598bb3938e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.516669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.516602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1966b53-b777-4ad3-831e-0598bb3938e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.517426 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.517396 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1966b53-b777-4ad3-831e-0598bb3938e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.519304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.519193 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.519304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.519204 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.519304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.519258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.519304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.519288 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.519304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.519264 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.519550 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.519421 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.519550 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.519519 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d1966b53-b777-4ad3-831e-0598bb3938e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.519787 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.519770 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d1966b53-b777-4ad3-831e-0598bb3938e9-web-config\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.520923 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.520906 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d1966b53-b777-4ad3-831e-0598bb3938e9-config-out\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.525572 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.525553 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4c5d\" (UniqueName: \"kubernetes.io/projected/d1966b53-b777-4ad3-831e-0598bb3938e9-kube-api-access-l4c5d\") pod \"alertmanager-main-0\" (UID: \"d1966b53-b777-4ad3-831e-0598bb3938e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.626446 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.626416 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 18:06:07.914900 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:07.914844 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 18:06:07.920360 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:06:07.920330 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1966b53_b777_4ad3_831e_0598bb3938e9.slice/crio-7ac9ad97b7779f9d4b0e10c0ca29b2a5f5513f2602d26f41a4c6a57703d3fce6 WatchSource:0}: Error finding container 7ac9ad97b7779f9d4b0e10c0ca29b2a5f5513f2602d26f41a4c6a57703d3fce6: Status 404 returned error can't find the container with id 7ac9ad97b7779f9d4b0e10c0ca29b2a5f5513f2602d26f41a4c6a57703d3fce6 Sep 17 18:06:08.241159 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:08.241074 2582 generic.go:356] "Generic (PLEG): container finished" podID="d1966b53-b777-4ad3-831e-0598bb3938e9" containerID="52b8c4c3b6f3783ca9ce876812ae3c717295149ea7ec8bfb41f0a55a11178b29" exitCode=0 Sep 17 18:06:08.241159 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:08.241135 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1966b53-b777-4ad3-831e-0598bb3938e9","Type":"ContainerDied","Data":"52b8c4c3b6f3783ca9ce876812ae3c717295149ea7ec8bfb41f0a55a11178b29"} Sep 17 18:06:08.241331 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:08.241199 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1966b53-b777-4ad3-831e-0598bb3938e9","Type":"ContainerStarted","Data":"7ac9ad97b7779f9d4b0e10c0ca29b2a5f5513f2602d26f41a4c6a57703d3fce6"} Sep 17 18:06:09.248687 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.248656 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1966b53-b777-4ad3-831e-0598bb3938e9","Type":"ContainerStarted","Data":"fbe1c99adfea1473bd95282af460f6da0c6188d098f87fdb57bc029fabb634e8"} Sep 17 18:06:09.249093 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.248716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1966b53-b777-4ad3-831e-0598bb3938e9","Type":"ContainerStarted","Data":"d7ee393d391a344ba42966aa988d84c91b2908b6be8521aeed01d6b381810611"} Sep 17 18:06:09.249093 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.248745 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1966b53-b777-4ad3-831e-0598bb3938e9","Type":"ContainerStarted","Data":"f667054e78cba9be6972be9af77c5646e495dc669d88b464f36c8793e4378eeb"} Sep 17 18:06:09.249093 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.248772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1966b53-b777-4ad3-831e-0598bb3938e9","Type":"ContainerStarted","Data":"b4367f5237982a9bba22307320e4b573ef8d0e8c1cb7e9c8eafdec31a5d950a9"} Sep 17 18:06:09.249093 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.248795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1966b53-b777-4ad3-831e-0598bb3938e9","Type":"ContainerStarted","Data":"f5c3f43d20df330b8c7556e388a0df884815b99c76e9d97bd720858251420ed6"} Sep 17 18:06:09.249093 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.248818 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d1966b53-b777-4ad3-831e-0598bb3938e9","Type":"ContainerStarted","Data":"c0ee99f3d100b7c025839cd1b9217bcb288ec05bf076dde8a95627353034adcb"} Sep 17 18:06:09.284595 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.284545 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.265192122 podStartE2EDuration="2.284525227s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.333105ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:09.282022277 +0000 UTC m=+180.345966234" watchObservedRunningTime="2026-09-17 18:06:09.284525227 +0000 UTC m=+180.348469185" Sep 17 18:06:09.818736 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.818697 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:06:09.819140 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.819106 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="prometheus" containerID="cri-o://eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" gracePeriod=600 Sep 17 18:06:09.819245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.819146 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy" containerID="cri-o://d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" gracePeriod=600 Sep 17 18:06:09.819245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.819131 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy-web" containerID="cri-o://e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" gracePeriod=600 Sep 17 18:06:09.819245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.819201 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="config-reloader" containerID="cri-o://713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" gracePeriod=600 Sep 17 18:06:09.819388 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.819159 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="thanos-sidecar" containerID="cri-o://8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" gracePeriod=600 Sep 17 18:06:09.819388 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:09.819113 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" gracePeriod=600 Sep 17 18:06:10.051631 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.051607 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.139104 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139017 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzcbn\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-kube-api-access-qzcbn\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139104 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139087 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-kube-rbac-proxy\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139115 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-serving-certs-ca-bundle\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139175 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-grpc-tls\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139200 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-0\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139232 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-thanos-prometheus-http-client-file\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139260 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-2\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139300 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-tls\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139321 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-trusted-ca-bundle\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139366 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139399 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-1\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139438 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139456 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-metrics-client-ca\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139478 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config-out\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139500 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-db\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139569 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-metrics-client-certs\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139601 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139638 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-web-config\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.139678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139662 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-tls-assets\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.140256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139689 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-kubelet-serving-ca-bundle\") pod \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\" (UID: \"1d78faaa-2cc2-4511-beb4-34bfd88e098f\") " Sep 17 18:06:10.140256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139718 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:10.140256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.139985 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.140256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.140241 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:10.140452 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.140358 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:10.140958 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.140666 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:10.140958 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.140884 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:10.141699 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.141656 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-kube-api-access-qzcbn" (OuterVolumeSpecName: "kube-api-access-qzcbn") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "kube-api-access-qzcbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:06:10.142145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.142117 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.142747 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.142717 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.143380 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.143335 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:06:10.143380 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.143334 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:10.143380 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.143366 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config-out" (OuterVolumeSpecName: "config-out") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:06:10.143576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.143394 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.143576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.143557 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:06:10.143804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.143773 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.144018 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.143960 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.144423 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.144395 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.145208 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.145187 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.145657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.145632 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config" (OuterVolumeSpecName: "config") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.145749 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.145731 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:06:10.154684 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.154662 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-web-config" (OuterVolumeSpecName: "web-config") pod "1d78faaa-2cc2-4511-beb4-34bfd88e098f" (UID: "1d78faaa-2cc2-4511-beb4-34bfd88e098f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:06:10.240593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240551 2582 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240589 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240600 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240610 2582 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240619 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240629 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240639 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-metrics-client-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240648 2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-config-out\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240656 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-db\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240665 2582 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-metrics-client-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240673 2582 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240681 2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-web-config\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240689 2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-tls-assets\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240696 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240704 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzcbn\" (UniqueName: \"kubernetes.io/projected/1d78faaa-2cc2-4511-beb4-34bfd88e098f-kube-api-access-qzcbn\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240712 2582 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-kube-rbac-proxy\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240720 2582 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240728 2582 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1d78faaa-2cc2-4511-beb4-34bfd88e098f-secret-grpc-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.240796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.240738 2582 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1d78faaa-2cc2-4511-beb4-34bfd88e098f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:06:10.256028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.255992 2582 generic.go:356] "Generic (PLEG): container finished" podID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" exitCode=0 Sep 17 18:06:10.256028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256016 2582 generic.go:356] "Generic (PLEG): container finished" podID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" exitCode=0 Sep 17 18:06:10.256028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256022 2582 generic.go:356] "Generic (PLEG): container finished" podID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" exitCode=0 Sep 17 18:06:10.256028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256029 2582 generic.go:356] "Generic (PLEG): container finished" podID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" exitCode=0 Sep 17 18:06:10.256028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256034 2582 generic.go:356] "Generic (PLEG): container finished" podID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" exitCode=0 Sep 17 18:06:10.256028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256039 2582 generic.go:356] "Generic (PLEG): container finished" podID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" exitCode=0 Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256077 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256074 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerDied","Data":"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c"} Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256149 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerDied","Data":"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f"} Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256182 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerDied","Data":"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4"} Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256209 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerDied","Data":"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6"} Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256215 2582 scope.go:117] "RemoveContainer" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256232 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerDied","Data":"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8"} Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerDied","Data":"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff"} Sep 17 18:06:10.256652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.256281 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1d78faaa-2cc2-4511-beb4-34bfd88e098f","Type":"ContainerDied","Data":"385f77bc5dc2926ec10bec751fab49745ab92d0c92b63c58895a6e020fe91aaa"} Sep 17 18:06:10.264702 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.264686 2582 scope.go:117] "RemoveContainer" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" Sep 17 18:06:10.274068 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.274047 2582 scope.go:117] "RemoveContainer" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" Sep 17 18:06:10.281137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.281118 2582 scope.go:117] "RemoveContainer" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" Sep 17 18:06:10.281939 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.281915 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:06:10.287362 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.287342 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:06:10.289534 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.289521 2582 scope.go:117] "RemoveContainer" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" Sep 17 18:06:10.296410 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.296392 2582 scope.go:117] "RemoveContainer" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" Sep 17 18:06:10.303432 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.303416 2582 scope.go:117] "RemoveContainer" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" Sep 17 18:06:10.310426 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.310407 2582 scope.go:117] "RemoveContainer" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" Sep 17 18:06:10.310711 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:10.310693 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": container with ID starting with 7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c not found: ID does not exist" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" Sep 17 18:06:10.310772 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.310720 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c"} err="failed to get container status \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": rpc error: code = NotFound desc = could not find container \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": container with ID starting with 7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c not found: ID does not exist" Sep 17 18:06:10.310772 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.310738 2582 scope.go:117] "RemoveContainer" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" Sep 17 18:06:10.310997 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:10.310980 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": container with ID starting with d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f not found: ID does not exist" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" Sep 17 18:06:10.311066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.311006 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f"} err="failed to get container status \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": rpc error: code = NotFound desc = could not find container \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": container with ID starting with d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f not found: ID does not exist" Sep 17 18:06:10.311066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.311028 2582 scope.go:117] "RemoveContainer" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" Sep 17 18:06:10.311287 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:10.311268 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": container with ID starting with e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4 not found: ID does not exist" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" Sep 17 18:06:10.311324 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.311295 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4"} err="failed to get container status \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": rpc error: code = NotFound desc = could not find container \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": container with ID starting with e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4 not found: ID does not exist" Sep 17 18:06:10.311324 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.311311 2582 scope.go:117] "RemoveContainer" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" Sep 17 18:06:10.311534 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:10.311520 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": container with ID starting with 8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6 not found: ID does not exist" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" Sep 17 18:06:10.311576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.311538 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6"} err="failed to get container status \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": rpc error: code = NotFound desc = could not find container \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": container with ID starting with 8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6 not found: ID does not exist" Sep 17 18:06:10.311576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.311551 2582 scope.go:117] "RemoveContainer" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" Sep 17 18:06:10.311757 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:10.311741 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": container with ID starting with 713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8 not found: ID does not exist" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" Sep 17 18:06:10.311817 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.311763 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8"} err="failed to get container status \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": rpc error: code = NotFound desc = could not find container \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": container with ID starting with 713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8 not found: ID does not exist" Sep 17 18:06:10.311817 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.311782 2582 scope.go:117] "RemoveContainer" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" Sep 17 18:06:10.312039 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:10.312024 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": container with ID starting with eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff not found: ID does not exist" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" Sep 17 18:06:10.312104 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.312043 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff"} err="failed to get container status \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": rpc error: code = NotFound desc = could not find container \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": container with ID starting with eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff not found: ID does not exist" Sep 17 18:06:10.312104 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.312056 2582 scope.go:117] "RemoveContainer" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" Sep 17 18:06:10.312294 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:06:10.312279 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": container with ID starting with 19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647 not found: ID does not exist" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" Sep 17 18:06:10.312330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.312297 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647"} err="failed to get container status \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": rpc error: code = NotFound desc = could not find container \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": container with ID starting with 19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647 not found: ID does not exist" Sep 17 18:06:10.312330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.312310 2582 scope.go:117] "RemoveContainer" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" Sep 17 18:06:10.312551 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.312532 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c"} err="failed to get container status \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": rpc error: code = NotFound desc = could not find container \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": container with ID starting with 7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c not found: ID does not exist" Sep 17 18:06:10.312589 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.312552 2582 scope.go:117] "RemoveContainer" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" Sep 17 18:06:10.312790 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.312775 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f"} err="failed to get container status \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": rpc error: code = NotFound desc = could not find container \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": container with ID starting with d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f not found: ID does not exist" Sep 17 18:06:10.312790 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.312790 2582 scope.go:117] "RemoveContainer" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" Sep 17 18:06:10.313029 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313015 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4"} err="failed to get container status \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": rpc error: code = NotFound desc = could not find container \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": container with ID starting with e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4 not found: ID does not exist" Sep 17 18:06:10.313073 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313030 2582 scope.go:117] "RemoveContainer" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" Sep 17 18:06:10.313241 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313225 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6"} err="failed to get container status \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": rpc error: code = NotFound desc = could not find container \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": container with ID starting with 8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6 not found: ID does not exist" Sep 17 18:06:10.313283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313241 2582 scope.go:117] "RemoveContainer" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" Sep 17 18:06:10.313458 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313440 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8"} err="failed to get container status \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": rpc error: code = NotFound desc = could not find container \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": container with ID starting with 713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8 not found: ID does not exist" Sep 17 18:06:10.313497 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313459 2582 scope.go:117] "RemoveContainer" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" Sep 17 18:06:10.313679 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313662 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff"} err="failed to get container status \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": rpc error: code = NotFound desc = could not find container \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": container with ID starting with eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff not found: ID does not exist" Sep 17 18:06:10.313679 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313678 2582 scope.go:117] "RemoveContainer" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" Sep 17 18:06:10.313913 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313898 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647"} err="failed to get container status \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": rpc error: code = NotFound desc = could not find container \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": container with ID starting with 19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647 not found: ID does not exist" Sep 17 18:06:10.313913 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.313913 2582 scope.go:117] "RemoveContainer" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" Sep 17 18:06:10.314162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.314145 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c"} err="failed to get container status \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": rpc error: code = NotFound desc = could not find container \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": container with ID starting with 7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c not found: ID does not exist" Sep 17 18:06:10.314204 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.314163 2582 scope.go:117] "RemoveContainer" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" Sep 17 18:06:10.314399 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.314384 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f"} err="failed to get container status \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": rpc error: code = NotFound desc = could not find container \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": container with ID starting with d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f not found: ID does not exist" Sep 17 18:06:10.314399 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.314397 2582 scope.go:117] "RemoveContainer" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" Sep 17 18:06:10.314639 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.314620 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4"} err="failed to get container status \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": rpc error: code = NotFound desc = could not find container \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": container with ID starting with e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4 not found: ID does not exist" Sep 17 18:06:10.314682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.314641 2582 scope.go:117] "RemoveContainer" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" Sep 17 18:06:10.314905 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.314886 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6"} err="failed to get container status \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": rpc error: code = NotFound desc = could not find container \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": container with ID starting with 8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6 not found: ID does not exist" Sep 17 18:06:10.314983 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.314908 2582 scope.go:117] "RemoveContainer" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" Sep 17 18:06:10.315166 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.315148 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8"} err="failed to get container status \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": rpc error: code = NotFound desc = could not find container \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": container with ID starting with 713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8 not found: ID does not exist" Sep 17 18:06:10.315223 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.315167 2582 scope.go:117] "RemoveContainer" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" Sep 17 18:06:10.315364 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.315348 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff"} err="failed to get container status \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": rpc error: code = NotFound desc = could not find container \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": container with ID starting with eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff not found: ID does not exist" Sep 17 18:06:10.315420 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.315364 2582 scope.go:117] "RemoveContainer" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" Sep 17 18:06:10.315600 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.315583 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647"} err="failed to get container status \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": rpc error: code = NotFound desc = could not find container \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": container with ID starting with 19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647 not found: ID does not exist" Sep 17 18:06:10.315600 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.315600 2582 scope.go:117] "RemoveContainer" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" Sep 17 18:06:10.315799 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.315780 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c"} err="failed to get container status \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": rpc error: code = NotFound desc = could not find container \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": container with ID starting with 7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c not found: ID does not exist" Sep 17 18:06:10.315874 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.315800 2582 scope.go:117] "RemoveContainer" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" Sep 17 18:06:10.316105 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316088 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f"} err="failed to get container status \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": rpc error: code = NotFound desc = could not find container \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": container with ID starting with d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f not found: ID does not exist" Sep 17 18:06:10.316173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316106 2582 scope.go:117] "RemoveContainer" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" Sep 17 18:06:10.316329 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316311 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4"} err="failed to get container status \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": rpc error: code = NotFound desc = could not find container \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": container with ID starting with e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4 not found: ID does not exist" Sep 17 18:06:10.316371 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316330 2582 scope.go:117] "RemoveContainer" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" Sep 17 18:06:10.316564 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316546 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6"} err="failed to get container status \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": rpc error: code = NotFound desc = could not find container \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": container with ID starting with 8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6 not found: ID does not exist" Sep 17 18:06:10.316630 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316565 2582 scope.go:117] "RemoveContainer" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" Sep 17 18:06:10.316778 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316761 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8"} err="failed to get container status \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": rpc error: code = NotFound desc = could not find container \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": container with ID starting with 713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8 not found: ID does not exist" Sep 17 18:06:10.316820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316779 2582 scope.go:117] "RemoveContainer" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" Sep 17 18:06:10.317015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.316991 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff"} err="failed to get container status \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": rpc error: code = NotFound desc = could not find container \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": container with ID starting with eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff not found: ID does not exist" Sep 17 18:06:10.317015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317013 2582 scope.go:117] "RemoveContainer" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" Sep 17 18:06:10.317197 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317178 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647"} err="failed to get container status \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": rpc error: code = NotFound desc = could not find container \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": container with ID starting with 19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647 not found: ID does not exist" Sep 17 18:06:10.317249 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317197 2582 scope.go:117] "RemoveContainer" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" Sep 17 18:06:10.317389 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317373 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c"} err="failed to get container status \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": rpc error: code = NotFound desc = could not find container \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": container with ID starting with 7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c not found: ID does not exist" Sep 17 18:06:10.317426 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317390 2582 scope.go:117] "RemoveContainer" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" Sep 17 18:06:10.317619 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317601 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f"} err="failed to get container status \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": rpc error: code = NotFound desc = could not find container \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": container with ID starting with d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f not found: ID does not exist" Sep 17 18:06:10.317619 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317617 2582 scope.go:117] "RemoveContainer" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" Sep 17 18:06:10.317805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317789 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4"} err="failed to get container status \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": rpc error: code = NotFound desc = could not find container \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": container with ID starting with e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4 not found: ID does not exist" Sep 17 18:06:10.317853 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.317805 2582 scope.go:117] "RemoveContainer" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" Sep 17 18:06:10.318055 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318038 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6"} err="failed to get container status \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": rpc error: code = NotFound desc = could not find container \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": container with ID starting with 8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6 not found: ID does not exist" Sep 17 18:06:10.318055 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318054 2582 scope.go:117] "RemoveContainer" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" Sep 17 18:06:10.318252 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318237 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8"} err="failed to get container status \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": rpc error: code = NotFound desc = could not find container \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": container with ID starting with 713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8 not found: ID does not exist" Sep 17 18:06:10.318302 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318253 2582 scope.go:117] "RemoveContainer" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" Sep 17 18:06:10.318439 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318424 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff"} err="failed to get container status \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": rpc error: code = NotFound desc = could not find container \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": container with ID starting with eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff not found: ID does not exist" Sep 17 18:06:10.318439 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318438 2582 scope.go:117] "RemoveContainer" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" Sep 17 18:06:10.318678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318663 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647"} err="failed to get container status \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": rpc error: code = NotFound desc = could not find container \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": container with ID starting with 19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647 not found: ID does not exist" Sep 17 18:06:10.318678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318678 2582 scope.go:117] "RemoveContainer" containerID="7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c" Sep 17 18:06:10.318940 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318924 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c"} err="failed to get container status \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": rpc error: code = NotFound desc = could not find container \"7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c\": container with ID starting with 7a682fb8d99fcd3532124593b4ced7283c90bc5e66a3590a6bd05e690546122c not found: ID does not exist" Sep 17 18:06:10.318940 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.318941 2582 scope.go:117] "RemoveContainer" containerID="d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f" Sep 17 18:06:10.319145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319125 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f"} err="failed to get container status \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": rpc error: code = NotFound desc = could not find container \"d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f\": container with ID starting with d89cd6afae0257f2e2ffa2760e614af63516595278c12cf40ab65a74b133e75f not found: ID does not exist" Sep 17 18:06:10.319187 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319146 2582 scope.go:117] "RemoveContainer" containerID="e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4" Sep 17 18:06:10.319368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319349 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4"} err="failed to get container status \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": rpc error: code = NotFound desc = could not find container \"e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4\": container with ID starting with e25b11ef7b46ea2a0b7d1ba6ad62dfa7cf6887cab284cace4e375a9c2a63fef4 not found: ID does not exist" Sep 17 18:06:10.319413 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319370 2582 scope.go:117] "RemoveContainer" containerID="8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6" Sep 17 18:06:10.319606 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319589 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6"} err="failed to get container status \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": rpc error: code = NotFound desc = could not find container \"8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6\": container with ID starting with 8c223c065bb40e9c6bf3d8c5fd2770a8a10fdd0ad4d527d11e5f14636dd9a1c6 not found: ID does not exist" Sep 17 18:06:10.319606 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319605 2582 scope.go:117] "RemoveContainer" containerID="713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8" Sep 17 18:06:10.319772 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319754 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8"} err="failed to get container status \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": rpc error: code = NotFound desc = could not find container \"713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8\": container with ID starting with 713f993cf922aa0ffd9257be82c18b363d3f5cd2e31bfbfb77832c69e25510e8 not found: ID does not exist" Sep 17 18:06:10.319772 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319770 2582 scope.go:117] "RemoveContainer" containerID="eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff" Sep 17 18:06:10.319976 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319957 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff"} err="failed to get container status \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": rpc error: code = NotFound desc = could not find container \"eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff\": container with ID starting with eff77efaa79ffdd34802b600602ab7edb4d4c9d7a5c768465563e865d9636aff not found: ID does not exist" Sep 17 18:06:10.320023 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.319977 2582 scope.go:117] "RemoveContainer" containerID="19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647" Sep 17 18:06:10.320213 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.320196 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647"} err="failed to get container status \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": rpc error: code = NotFound desc = could not find container \"19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647\": container with ID starting with 19c0fc721349a3fdfc99a83a1f70958d5e5404f95aa5b37cd9c612c231fba647 not found: ID does not exist" Sep 17 18:06:10.322453 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322433 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:06:10.322724 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322712 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="thanos-sidecar" Sep 17 18:06:10.322782 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322727 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="thanos-sidecar" Sep 17 18:06:10.322782 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322743 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="config-reloader" Sep 17 18:06:10.322782 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322760 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="config-reloader" Sep 17 18:06:10.322782 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322771 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="init-config-reloader" Sep 17 18:06:10.322782 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322777 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="init-config-reloader" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322785 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy-web" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322791 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy-web" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322800 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322806 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322812 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy-thanos" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322818 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy-thanos" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322827 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="prometheus" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322832 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="prometheus" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322888 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy-web" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322898 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322905 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="kube-rbac-proxy-thanos" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322912 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="thanos-sidecar" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322918 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="prometheus" Sep 17 18:06:10.322941 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.322924 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" containerName="config-reloader" Sep 17 18:06:10.326557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.326539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.336926 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.336908 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 17 18:06:10.336926 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.336920 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 17 18:06:10.337089 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.336909 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 17 18:06:10.337089 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.336932 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 17 18:06:10.337170 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.337152 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 17 18:06:10.337214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.337195 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 17 18:06:10.337310 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.337291 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1vi0cqopbgbmr\"" Sep 17 18:06:10.337437 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.337425 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mmgbt\"" Sep 17 18:06:10.337481 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.337449 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 17 18:06:10.337617 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.337603 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 17 18:06:10.337670 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.337608 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 17 18:06:10.337956 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.337942 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 17 18:06:10.338403 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.338388 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 17 18:06:10.339224 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.339210 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 17 18:06:10.340723 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.340698 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 17 18:06:10.342355 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.342338 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 17 18:06:10.348471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.348452 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:06:10.442015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.441987 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2774be6c-12e8-4b9a-a46f-edbda1d65655-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2774be6c-12e8-4b9a-a46f-edbda1d65655-config-out\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442047 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-web-config\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442136 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-config\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442206 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442225 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442251 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442272 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442298 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442372 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442394 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442695 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442695 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.442695 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.442509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fkf\" (UniqueName: \"kubernetes.io/projected/2774be6c-12e8-4b9a-a46f-edbda1d65655-kube-api-access-98fkf\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.543843 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.543812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2774be6c-12e8-4b9a-a46f-edbda1d65655-config-out\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.543854 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.543908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.543925 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-web-config\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.543941 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-config\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.543958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.543990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544018 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544107 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544154 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544228 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544366 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544856 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544347 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544856 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fkf\" (UniqueName: \"kubernetes.io/projected/2774be6c-12e8-4b9a-a46f-edbda1d65655-kube-api-access-98fkf\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.544856 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2774be6c-12e8-4b9a-a46f-edbda1d65655-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.545044 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.544944 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.546762 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.545558 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.546762 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.546210 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.546762 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.546654 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2774be6c-12e8-4b9a-a46f-edbda1d65655-config-out\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.546762 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.546947 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.546762 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.546982 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-web-config\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.547410 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.547133 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2774be6c-12e8-4b9a-a46f-edbda1d65655-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.547410 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.547192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.547410 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.547240 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.547814 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.547784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.547814 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.547809 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.548575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.548544 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.549108 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.549084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-config\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.549527 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.549464 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.549745 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.549721 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.549836 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.549818 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.550457 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.550437 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2774be6c-12e8-4b9a-a46f-edbda1d65655-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.550556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.550539 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2774be6c-12e8-4b9a-a46f-edbda1d65655-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.554343 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.554320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fkf\" (UniqueName: \"kubernetes.io/projected/2774be6c-12e8-4b9a-a46f-edbda1d65655-kube-api-access-98fkf\") pod \"prometheus-k8s-0\" (UID: \"2774be6c-12e8-4b9a-a46f-edbda1d65655\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.635137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.635108 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:06:10.937890 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:10.934679 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 17 18:06:10.939629 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:06:10.939601 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2774be6c_12e8_4b9a_a46f_edbda1d65655.slice/crio-4507bd35c85a9b1e8be36106357fcbcb681031cbe2c688671ca536ee2a243209 WatchSource:0}: Error finding container 4507bd35c85a9b1e8be36106357fcbcb681031cbe2c688671ca536ee2a243209: Status 404 returned error can't find the container with id 4507bd35c85a9b1e8be36106357fcbcb681031cbe2c688671ca536ee2a243209 Sep 17 18:06:11.261370 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:11.261334 2582 generic.go:356] "Generic (PLEG): container finished" podID="2774be6c-12e8-4b9a-a46f-edbda1d65655" containerID="15b4d81520467bf11e65b28b160fd9f7b907005c603b63c009705226d536b989" exitCode=0 Sep 17 18:06:11.261714 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:11.261420 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2774be6c-12e8-4b9a-a46f-edbda1d65655","Type":"ContainerDied","Data":"15b4d81520467bf11e65b28b160fd9f7b907005c603b63c009705226d536b989"} Sep 17 18:06:11.261714 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:11.261484 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2774be6c-12e8-4b9a-a46f-edbda1d65655","Type":"ContainerStarted","Data":"4507bd35c85a9b1e8be36106357fcbcb681031cbe2c688671ca536ee2a243209"} Sep 17 18:06:11.347099 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:11.347073 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d78faaa-2cc2-4511-beb4-34bfd88e098f" path="/var/lib/kubelet/pods/1d78faaa-2cc2-4511-beb4-34bfd88e098f/volumes" Sep 17 18:06:12.271173 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:12.271135 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2774be6c-12e8-4b9a-a46f-edbda1d65655","Type":"ContainerStarted","Data":"7ae363b8f05afe6f5d2092ef07e0d3a33a9080493254a1b32873449cd8e20b00"} Sep 17 18:06:12.271557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:12.271228 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2774be6c-12e8-4b9a-a46f-edbda1d65655","Type":"ContainerStarted","Data":"c5cb65d291a040fd5083b8ae5c9d1b395f6ceb70cfd3b8bea426c5104240ee08"} Sep 17 18:06:12.271557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:12.271255 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2774be6c-12e8-4b9a-a46f-edbda1d65655","Type":"ContainerStarted","Data":"afaf426c57122b356a84b27a78708b5aaa2b6035f7128c20dee454ec74e586e8"} Sep 17 18:06:12.271557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:12.271278 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2774be6c-12e8-4b9a-a46f-edbda1d65655","Type":"ContainerStarted","Data":"1eb39916abcaa1232d0c2ac641d353616b5c8576b027af2401465adbf494cc9a"} Sep 17 18:06:12.271557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:12.271304 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2774be6c-12e8-4b9a-a46f-edbda1d65655","Type":"ContainerStarted","Data":"c1749baa43034e7d481075582726848f333f4524db9d1b3fc040ab4536abbdca"} Sep 17 18:06:12.271557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:12.271327 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2774be6c-12e8-4b9a-a46f-edbda1d65655","Type":"ContainerStarted","Data":"a60acac0cf80b9b264e70a94d84a1ed6d19110fd764a47fcf858cf7abef515b3"} Sep 17 18:06:12.310942 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:12.310889 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.292049949 podStartE2EDuration="2.310854784s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.804835ms" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:12.308545657 +0000 UTC m=+183.372489615" watchObservedRunningTime="2026-09-17 18:06:12.310854784 +0000 UTC m=+183.374798743" Sep 17 18:06:15.636145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:06:15.636106 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:07:10.636031 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:10.635996 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:07:10.651315 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:10.651291 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:07:11.515483 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:11.515458 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 17 18:07:42.933189 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:42.933103 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lvnt9"] Sep 17 18:07:42.936341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:42.936324 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lvnt9" Sep 17 18:07:42.939502 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:42.939483 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gzf9w\"" Sep 17 18:07:42.939753 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:42.939741 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 17 18:07:42.940421 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:42.940403 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 17 18:07:42.952257 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:42.952233 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lvnt9"] Sep 17 18:07:43.015636 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.015605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8493fdd1-8a08-42f1-b509-abf80409663b-bound-sa-token\") pod \"cert-manager-55b4d954d5-lvnt9\" (UID: \"8493fdd1-8a08-42f1-b509-abf80409663b\") " pod="cert-manager/cert-manager-55b4d954d5-lvnt9" Sep 17 18:07:43.015787 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.015651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxjnh\" (UniqueName: \"kubernetes.io/projected/8493fdd1-8a08-42f1-b509-abf80409663b-kube-api-access-cxjnh\") pod \"cert-manager-55b4d954d5-lvnt9\" (UID: \"8493fdd1-8a08-42f1-b509-abf80409663b\") " pod="cert-manager/cert-manager-55b4d954d5-lvnt9" Sep 17 18:07:43.116843 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.116812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8493fdd1-8a08-42f1-b509-abf80409663b-bound-sa-token\") pod \"cert-manager-55b4d954d5-lvnt9\" (UID: \"8493fdd1-8a08-42f1-b509-abf80409663b\") " pod="cert-manager/cert-manager-55b4d954d5-lvnt9" Sep 17 18:07:43.117006 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.116855 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxjnh\" (UniqueName: \"kubernetes.io/projected/8493fdd1-8a08-42f1-b509-abf80409663b-kube-api-access-cxjnh\") pod \"cert-manager-55b4d954d5-lvnt9\" (UID: \"8493fdd1-8a08-42f1-b509-abf80409663b\") " pod="cert-manager/cert-manager-55b4d954d5-lvnt9" Sep 17 18:07:43.140570 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.140541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8493fdd1-8a08-42f1-b509-abf80409663b-bound-sa-token\") pod \"cert-manager-55b4d954d5-lvnt9\" (UID: \"8493fdd1-8a08-42f1-b509-abf80409663b\") " pod="cert-manager/cert-manager-55b4d954d5-lvnt9" Sep 17 18:07:43.140688 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.140625 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxjnh\" (UniqueName: \"kubernetes.io/projected/8493fdd1-8a08-42f1-b509-abf80409663b-kube-api-access-cxjnh\") pod \"cert-manager-55b4d954d5-lvnt9\" (UID: \"8493fdd1-8a08-42f1-b509-abf80409663b\") " pod="cert-manager/cert-manager-55b4d954d5-lvnt9" Sep 17 18:07:43.257035 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.256962 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lvnt9" Sep 17 18:07:43.535439 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.535416 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lvnt9"] Sep 17 18:07:43.537935 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:07:43.537904 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8493fdd1_8a08_42f1_b509_abf80409663b.slice/crio-c861e928089084655dd82e3bda5f6a19ef2f60888889223e2dabe15087b99bf7 WatchSource:0}: Error finding container c861e928089084655dd82e3bda5f6a19ef2f60888889223e2dabe15087b99bf7: Status 404 returned error can't find the container with id c861e928089084655dd82e3bda5f6a19ef2f60888889223e2dabe15087b99bf7 Sep 17 18:07:43.627513 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:43.627474 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lvnt9" event={"ID":"8493fdd1-8a08-42f1-b509-abf80409663b","Type":"ContainerStarted","Data":"c861e928089084655dd82e3bda5f6a19ef2f60888889223e2dabe15087b99bf7"} Sep 17 18:07:46.681834 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.681803 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf"] Sep 17 18:07:46.685005 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.684987 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" Sep 17 18:07:46.688511 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.688488 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 17 18:07:46.688622 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.688486 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 17 18:07:46.696352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.696333 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jhvrg\"" Sep 17 18:07:46.715195 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.715175 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf"] Sep 17 18:07:46.751737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.751714 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh9xd\" (UniqueName: \"kubernetes.io/projected/0b903663-c19e-409a-ad56-44268c75af1e-kube-api-access-fh9xd\") pod \"openshift-lws-operator-fd8ccff4c-tfmcf\" (UID: \"0b903663-c19e-409a-ad56-44268c75af1e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" Sep 17 18:07:46.751879 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.751780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b903663-c19e-409a-ad56-44268c75af1e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tfmcf\" (UID: \"0b903663-c19e-409a-ad56-44268c75af1e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" Sep 17 18:07:46.852737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.852705 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b903663-c19e-409a-ad56-44268c75af1e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tfmcf\" (UID: \"0b903663-c19e-409a-ad56-44268c75af1e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" Sep 17 18:07:46.852919 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.852759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh9xd\" (UniqueName: \"kubernetes.io/projected/0b903663-c19e-409a-ad56-44268c75af1e-kube-api-access-fh9xd\") pod \"openshift-lws-operator-fd8ccff4c-tfmcf\" (UID: \"0b903663-c19e-409a-ad56-44268c75af1e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" Sep 17 18:07:46.853092 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.853074 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b903663-c19e-409a-ad56-44268c75af1e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tfmcf\" (UID: \"0b903663-c19e-409a-ad56-44268c75af1e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" Sep 17 18:07:46.861910 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.861883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh9xd\" (UniqueName: \"kubernetes.io/projected/0b903663-c19e-409a-ad56-44268c75af1e-kube-api-access-fh9xd\") pod \"openshift-lws-operator-fd8ccff4c-tfmcf\" (UID: \"0b903663-c19e-409a-ad56-44268c75af1e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" Sep 17 18:07:46.994385 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:46.994292 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" Sep 17 18:07:47.283648 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:47.283581 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf"] Sep 17 18:07:47.286926 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:07:47.286895 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b903663_c19e_409a_ad56_44268c75af1e.slice/crio-60f3760a85bea056fd56dcfd6f46cf79f962f518650d10d189d927c0b07b9c14 WatchSource:0}: Error finding container 60f3760a85bea056fd56dcfd6f46cf79f962f518650d10d189d927c0b07b9c14: Status 404 returned error can't find the container with id 60f3760a85bea056fd56dcfd6f46cf79f962f518650d10d189d927c0b07b9c14 Sep 17 18:07:47.649207 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:47.649180 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" event={"ID":"0b903663-c19e-409a-ad56-44268c75af1e","Type":"ContainerStarted","Data":"60f3760a85bea056fd56dcfd6f46cf79f962f518650d10d189d927c0b07b9c14"} Sep 17 18:07:47.650649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:47.650625 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lvnt9" event={"ID":"8493fdd1-8a08-42f1-b509-abf80409663b","Type":"ContainerStarted","Data":"04a5417f35d27a4abfc37b65f6d2259adfb34db6a2756352f3107a066e078e5b"} Sep 17 18:07:47.676951 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:47.676902 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lvnt9" podStartSLOduration=2.517036144 podStartE2EDuration="5.676886812s" totalImagesPullingTime="3.159850668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:47.675691499 +0000 UTC m=+278.739635451" watchObservedRunningTime="2026-09-17 18:07:47.676886812 +0000 UTC m=+278.740830761" Sep 17 18:07:50.669174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:50.669142 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" event={"ID":"0b903663-c19e-409a-ad56-44268c75af1e","Type":"ContainerStarted","Data":"86e52e69f6faa11cb6731c8a5515854703ed5110c4830e0be231b82235481df5"} Sep 17 18:07:50.694559 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:07:50.694503 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tfmcf" podStartSLOduration=2.185965859 podStartE2EDuration="4.694486686s" totalImagesPullingTime="2.508520827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:50.69313168 +0000 UTC m=+281.757075637" watchObservedRunningTime="2026-09-17 18:07:50.694486686 +0000 UTC m=+281.758430644" Sep 17 18:08:07.213713 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.213677 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5"] Sep 17 18:08:07.217494 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.217476 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.220578 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.220555 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 18:08:07.220689 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.220614 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 18:08:07.221083 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.221068 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 18:08:07.221135 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.221072 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 18:08:07.221442 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.221422 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8977h\"" Sep 17 18:08:07.236451 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.236429 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5"] Sep 17 18:08:07.330443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.330410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrksx\" (UniqueName: \"kubernetes.io/projected/c735fac7-125f-41fc-ab73-3a51824c554a-kube-api-access-xrksx\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.330593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.330464 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c735fac7-125f-41fc-ab73-3a51824c554a-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.330593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.330528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c735fac7-125f-41fc-ab73-3a51824c554a-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.330593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.330547 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c735fac7-125f-41fc-ab73-3a51824c554a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.330714 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.330613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c735fac7-125f-41fc-ab73-3a51824c554a-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.431622 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.431586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrksx\" (UniqueName: \"kubernetes.io/projected/c735fac7-125f-41fc-ab73-3a51824c554a-kube-api-access-xrksx\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.431783 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.431641 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c735fac7-125f-41fc-ab73-3a51824c554a-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.431831 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.431810 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c735fac7-125f-41fc-ab73-3a51824c554a-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.431898 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.431855 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c735fac7-125f-41fc-ab73-3a51824c554a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.431944 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.431931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c735fac7-125f-41fc-ab73-3a51824c554a-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.432119 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.432097 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c735fac7-125f-41fc-ab73-3a51824c554a-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.434132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.434110 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c735fac7-125f-41fc-ab73-3a51824c554a-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.434336 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.434320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c735fac7-125f-41fc-ab73-3a51824c554a-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.434475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.434453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c735fac7-125f-41fc-ab73-3a51824c554a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.451388 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.451364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrksx\" (UniqueName: \"kubernetes.io/projected/c735fac7-125f-41fc-ab73-3a51824c554a-kube-api-access-xrksx\") pod \"opendatahub-operator-controller-manager-648c847c44-6h5l5\" (UID: \"c735fac7-125f-41fc-ab73-3a51824c554a\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.527984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.527911 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:07.814185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:07.814106 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5"] Sep 17 18:08:07.817239 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:08:07.817212 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc735fac7_125f_41fc_ab73_3a51824c554a.slice/crio-2dba3935520dbb34ec4594f3d56f0de6791403c6d099ee54758a77a7e2445c88 WatchSource:0}: Error finding container 2dba3935520dbb34ec4594f3d56f0de6791403c6d099ee54758a77a7e2445c88: Status 404 returned error can't find the container with id 2dba3935520dbb34ec4594f3d56f0de6791403c6d099ee54758a77a7e2445c88 Sep 17 18:08:08.744272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:08.744235 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" event={"ID":"c735fac7-125f-41fc-ab73-3a51824c554a","Type":"ContainerStarted","Data":"2dba3935520dbb34ec4594f3d56f0de6791403c6d099ee54758a77a7e2445c88"} Sep 17 18:08:11.205954 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:11.205930 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:08:11.209278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:11.209254 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:08:11.216933 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:11.216912 2582 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 18:08:11.233683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:11.233659 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:08:11.237073 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:11.237050 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:08:11.768288 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:11.768254 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" event={"ID":"c735fac7-125f-41fc-ab73-3a51824c554a","Type":"ContainerStarted","Data":"fde447c19748929550ef4a9c3954ce0e07b273e83543e677f9d1bf5b6e5a0e20"} Sep 17 18:08:12.061649 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:08:12.061624 2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc735fac7_125f_41fc_ab73_3a51824c554a.slice/crio-fde447c19748929550ef4a9c3954ce0e07b273e83543e677f9d1bf5b6e5a0e20.scope\": RecentStats: unable to find data in memory cache]" Sep 17 18:08:12.774168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:12.774136 2582 generic.go:356] "Generic (PLEG): container finished" podID="c735fac7-125f-41fc-ab73-3a51824c554a" containerID="fde447c19748929550ef4a9c3954ce0e07b273e83543e677f9d1bf5b6e5a0e20" exitCode=0 Sep 17 18:08:12.774554 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:12.774213 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" event={"ID":"c735fac7-125f-41fc-ab73-3a51824c554a","Type":"ContainerDied","Data":"fde447c19748929550ef4a9c3954ce0e07b273e83543e677f9d1bf5b6e5a0e20"} Sep 17 18:08:12.774722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:12.774706 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:08:13.781627 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:13.781588 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" event={"ID":"c735fac7-125f-41fc-ab73-3a51824c554a","Type":"ContainerStarted","Data":"a09e3ebdee6e4fa1fc160361bc2ad5d5878821473df497afc947ef159fb30aad"} Sep 17 18:08:13.782059 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:13.781727 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:13.816181 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:13.816130 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" podStartSLOduration=2.318643168 podStartE2EDuration="6.816115264s" totalImagesPullingTime="3.402208283s" totalInitContainerRuntime="1.095263813s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:13.813335613 +0000 UTC m=+304.877279570" watchObservedRunningTime="2026-09-17 18:08:13.816115264 +0000 UTC m=+304.880059222" Sep 17 18:08:24.793348 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:24.793313 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-6h5l5" Sep 17 18:08:28.322441 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.322407 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw"] Sep 17 18:08:28.324887 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.324857 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.329836 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.329806 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 18:08:28.330711 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.330692 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 17 18:08:28.330950 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.330933 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 17 18:08:28.350033 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.350002 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 18:08:28.352125 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.352100 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw"] Sep 17 18:08:28.424203 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.424164 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3626a029-7f14-49ac-a105-d51681cb58c2-tls-certs\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.424417 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.424248 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3626a029-7f14-49ac-a105-d51681cb58c2-tmp\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.424417 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.424302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bpt\" (UniqueName: \"kubernetes.io/projected/3626a029-7f14-49ac-a105-d51681cb58c2-kube-api-access-w4bpt\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.525247 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.525206 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bpt\" (UniqueName: \"kubernetes.io/projected/3626a029-7f14-49ac-a105-d51681cb58c2-kube-api-access-w4bpt\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.525411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.525278 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3626a029-7f14-49ac-a105-d51681cb58c2-tls-certs\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.525411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.525320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3626a029-7f14-49ac-a105-d51681cb58c2-tmp\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.527564 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.527531 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3626a029-7f14-49ac-a105-d51681cb58c2-tmp\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.527749 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.527727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3626a029-7f14-49ac-a105-d51681cb58c2-tls-certs\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.546454 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.546424 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bpt\" (UniqueName: \"kubernetes.io/projected/3626a029-7f14-49ac-a105-d51681cb58c2-kube-api-access-w4bpt\") pod \"kube-auth-proxy-7bc8cbff6f-w2rcw\" (UID: \"3626a029-7f14-49ac-a105-d51681cb58c2\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.635639 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.635556 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" Sep 17 18:08:28.956598 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:28.956559 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw"] Sep 17 18:08:28.972809 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:08:28.972781 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3626a029_7f14_49ac_a105_d51681cb58c2.slice/crio-ef9f0a438f096e3477f3419a18b9b9be641e43ba3453988dfa241dd4c3b21e2c WatchSource:0}: Error finding container ef9f0a438f096e3477f3419a18b9b9be641e43ba3453988dfa241dd4c3b21e2c: Status 404 returned error can't find the container with id ef9f0a438f096e3477f3419a18b9b9be641e43ba3453988dfa241dd4c3b21e2c Sep 17 18:08:29.861231 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:29.861181 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" event={"ID":"3626a029-7f14-49ac-a105-d51681cb58c2","Type":"ContainerStarted","Data":"ef9f0a438f096e3477f3419a18b9b9be641e43ba3453988dfa241dd4c3b21e2c"} Sep 17 18:08:30.709595 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.709562 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5"] Sep 17 18:08:30.715510 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.714467 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.718152 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.718077 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 17 18:08:30.719079 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.719027 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 18:08:30.719079 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.719056 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 17 18:08:30.719278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.719056 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 17 18:08:30.719278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.719057 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-474z2\"" Sep 17 18:08:30.738961 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.738934 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5"] Sep 17 18:08:30.847113 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.847080 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.847278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.847119 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.847278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.847154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.847278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.847211 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.847433 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.847287 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.847433 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.847321 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.847433 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.847347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8r8\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-kube-api-access-6k8r8\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.948272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.948238 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.948722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.948290 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.948722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.948322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8r8\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-kube-api-access-6k8r8\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.948722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.948412 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.948722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.948444 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.948722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.948479 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.948722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.948509 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.950270 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.950197 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.952080 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.952054 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.955303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.954541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.955303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.955227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.958876 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.957927 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.960972 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.960916 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:30.961998 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:30.961957 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8r8\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-kube-api-access-6k8r8\") pod \"istiod-openshift-gateway-7b7995d5cd-57dw5\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:31.027822 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:31.027789 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:32.244208 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:32.244186 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5"] Sep 17 18:08:32.246759 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:08:32.246729 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f04729e_1f1d_47ef_a0db_6f47ecc2b8cf.slice/crio-083faf37f4aed37f2737066eed38fab7ae24970af518bf3bc85260182f750da5 WatchSource:0}: Error finding container 083faf37f4aed37f2737066eed38fab7ae24970af518bf3bc85260182f750da5: Status 404 returned error can't find the container with id 083faf37f4aed37f2737066eed38fab7ae24970af518bf3bc85260182f750da5 Sep 17 18:08:32.880497 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:32.880306 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" event={"ID":"3626a029-7f14-49ac-a105-d51681cb58c2","Type":"ContainerStarted","Data":"52eee96a946299c69dacf432669f2771ccd7efa58aec7c7afa3896d307727d7e"} Sep 17 18:08:32.881968 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:32.881940 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" event={"ID":"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf","Type":"ContainerStarted","Data":"083faf37f4aed37f2737066eed38fab7ae24970af518bf3bc85260182f750da5"} Sep 17 18:08:32.914658 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:32.914603 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-w2rcw" podStartSLOduration=1.867184374 podStartE2EDuration="4.914582987s" totalImagesPullingTime="3.047398613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:32.912763805 +0000 UTC m=+323.976707765" watchObservedRunningTime="2026-09-17 18:08:32.914582987 +0000 UTC m=+323.978526946" Sep 17 18:08:34.497561 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:34.497524 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 18:08:34.497802 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:34.497594 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 18:08:34.894185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:34.894088 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" event={"ID":"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf","Type":"ContainerStarted","Data":"d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0"} Sep 17 18:08:34.894356 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:34.894215 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:34.932276 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:34.932213 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" podStartSLOduration=2.683953432 podStartE2EDuration="4.93219352s" totalImagesPullingTime="2.248240088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:34.929969738 +0000 UTC m=+325.993913697" watchObservedRunningTime="2026-09-17 18:08:34.93219352 +0000 UTC m=+325.996137478" Sep 17 18:08:35.901396 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:35.901367 2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-57dw5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 17 18:08:35.901715 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:35.901433 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" podUID="4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:08:38.900509 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:38.900474 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:49.549329 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.549294 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt"] Sep 17 18:08:49.557697 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.557674 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv"] Sep 17 18:08:49.557842 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.557817 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:49.560989 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.560922 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:49.561137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.561107 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-l75jk\"" Sep 17 18:08:49.561345 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.561323 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 17 18:08:49.565200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.565181 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 17 18:08:49.565375 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.565241 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-skzd6\"" Sep 17 18:08:49.567630 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.567609 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt"] Sep 17 18:08:49.572929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.572904 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv"] Sep 17 18:08:49.607202 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.607174 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/247b1774-3590-4aa0-b528-ab6f38a22d7a-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdzjt\" (UID: \"247b1774-3590-4aa0-b528-ab6f38a22d7a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:49.607339 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.607215 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nppdf\" (UniqueName: \"kubernetes.io/projected/247b1774-3590-4aa0-b528-ab6f38a22d7a-kube-api-access-nppdf\") pod \"kserve-controller-manager-d6f4b65b8-bdzjt\" (UID: \"247b1774-3590-4aa0-b528-ab6f38a22d7a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:49.607339 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.607241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rftjv\" (UID: \"083acc7b-dc74-462f-b658-3a69fc43b013\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:49.607339 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.607283 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9tw\" (UniqueName: \"kubernetes.io/projected/083acc7b-dc74-462f-b658-3a69fc43b013-kube-api-access-wz9tw\") pod \"llmisvc-controller-manager-777c6bd7d5-rftjv\" (UID: \"083acc7b-dc74-462f-b658-3a69fc43b013\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:49.707666 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.707637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/247b1774-3590-4aa0-b528-ab6f38a22d7a-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdzjt\" (UID: \"247b1774-3590-4aa0-b528-ab6f38a22d7a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:49.707666 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.707670 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nppdf\" (UniqueName: \"kubernetes.io/projected/247b1774-3590-4aa0-b528-ab6f38a22d7a-kube-api-access-nppdf\") pod \"kserve-controller-manager-d6f4b65b8-bdzjt\" (UID: \"247b1774-3590-4aa0-b528-ab6f38a22d7a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:49.707923 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.707686 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rftjv\" (UID: \"083acc7b-dc74-462f-b658-3a69fc43b013\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:49.707923 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.707701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9tw\" (UniqueName: \"kubernetes.io/projected/083acc7b-dc74-462f-b658-3a69fc43b013-kube-api-access-wz9tw\") pod \"llmisvc-controller-manager-777c6bd7d5-rftjv\" (UID: \"083acc7b-dc74-462f-b658-3a69fc43b013\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:49.707923 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:08:49.707771 2582 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 17 18:08:49.707923 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:08:49.707797 2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 17 18:08:49.707923 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:08:49.707837 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/247b1774-3590-4aa0-b528-ab6f38a22d7a-cert podName:247b1774-3590-4aa0-b528-ab6f38a22d7a nodeName:}" failed. No retries permitted until 2026-09-17 18:08:50.20782141 +0000 UTC m=+341.271765346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/247b1774-3590-4aa0-b528-ab6f38a22d7a-cert") pod "kserve-controller-manager-d6f4b65b8-bdzjt" (UID: "247b1774-3590-4aa0-b528-ab6f38a22d7a") : secret "kserve-webhook-server-cert" not found Sep 17 18:08:49.707923 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:08:49.707851 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert podName:083acc7b-dc74-462f-b658-3a69fc43b013 nodeName:}" failed. No retries permitted until 2026-09-17 18:08:50.207845327 +0000 UTC m=+341.271789262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert") pod "llmisvc-controller-manager-777c6bd7d5-rftjv" (UID: "083acc7b-dc74-462f-b658-3a69fc43b013") : secret "llmisvc-webhook-server-cert" not found Sep 17 18:08:49.718686 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.718661 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nppdf\" (UniqueName: \"kubernetes.io/projected/247b1774-3590-4aa0-b528-ab6f38a22d7a-kube-api-access-nppdf\") pod \"kserve-controller-manager-d6f4b65b8-bdzjt\" (UID: \"247b1774-3590-4aa0-b528-ab6f38a22d7a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:49.720162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:49.720145 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9tw\" (UniqueName: \"kubernetes.io/projected/083acc7b-dc74-462f-b658-3a69fc43b013-kube-api-access-wz9tw\") pod \"llmisvc-controller-manager-777c6bd7d5-rftjv\" (UID: \"083acc7b-dc74-462f-b658-3a69fc43b013\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:50.211192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:50.211159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rftjv\" (UID: \"083acc7b-dc74-462f-b658-3a69fc43b013\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:50.211348 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:50.211300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/247b1774-3590-4aa0-b528-ab6f38a22d7a-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdzjt\" (UID: \"247b1774-3590-4aa0-b528-ab6f38a22d7a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:50.211348 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:08:50.211312 2582 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 17 18:08:50.211464 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:08:50.211375 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert podName:083acc7b-dc74-462f-b658-3a69fc43b013 nodeName:}" failed. No retries permitted until 2026-09-17 18:08:51.211355502 +0000 UTC m=+342.275299439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert") pod "llmisvc-controller-manager-777c6bd7d5-rftjv" (UID: "083acc7b-dc74-462f-b658-3a69fc43b013") : secret "llmisvc-webhook-server-cert" not found Sep 17 18:08:50.213975 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:50.213945 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/247b1774-3590-4aa0-b528-ab6f38a22d7a-cert\") pod \"kserve-controller-manager-d6f4b65b8-bdzjt\" (UID: \"247b1774-3590-4aa0-b528-ab6f38a22d7a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:50.471161 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:50.471074 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:50.790780 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:50.790696 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt"] Sep 17 18:08:50.793593 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:08:50.793561 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247b1774_3590_4aa0_b528_ab6f38a22d7a.slice/crio-7979f513c73cf7076ac01501673e5af59d1ce96165b9a0dc52eda225bb7245c0 WatchSource:0}: Error finding container 7979f513c73cf7076ac01501673e5af59d1ce96165b9a0dc52eda225bb7245c0: Status 404 returned error can't find the container with id 7979f513c73cf7076ac01501673e5af59d1ce96165b9a0dc52eda225bb7245c0 Sep 17 18:08:50.967138 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:50.967098 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" event={"ID":"247b1774-3590-4aa0-b528-ab6f38a22d7a","Type":"ContainerStarted","Data":"7979f513c73cf7076ac01501673e5af59d1ce96165b9a0dc52eda225bb7245c0"} Sep 17 18:08:51.220405 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.220378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rftjv\" (UID: \"083acc7b-dc74-462f-b658-3a69fc43b013\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:51.222693 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.222675 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/083acc7b-dc74-462f-b658-3a69fc43b013-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rftjv\" (UID: \"083acc7b-dc74-462f-b658-3a69fc43b013\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:51.298982 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.298950 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5"] Sep 17 18:08:51.299197 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.299174 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" podUID="4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" containerName="discovery" containerID="cri-o://d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0" gracePeriod=30 Sep 17 18:08:51.377346 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.377321 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:08:51.549068 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.549043 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:51.624131 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.624108 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k8r8\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-kube-api-access-6k8r8\") pod \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " Sep 17 18:08:51.624303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.624166 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-kubeconfig\") pod \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " Sep 17 18:08:51.624303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.624192 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-local-certs\") pod \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " Sep 17 18:08:51.624303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.624261 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-dns-cert\") pod \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " Sep 17 18:08:51.624303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.624296 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-token\") pod \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " Sep 17 18:08:51.624507 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.624325 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-cacerts\") pod \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " Sep 17 18:08:51.624507 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.624370 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-ca-configmap\") pod \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\" (UID: \"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf\") " Sep 17 18:08:51.626887 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.626823 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-kube-api-access-6k8r8" (OuterVolumeSpecName: "kube-api-access-6k8r8") pod "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" (UID: "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf"). InnerVolumeSpecName "kube-api-access-6k8r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:08:51.627017 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.626959 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" (UID: "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:08:51.628253 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.627966 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" (UID: "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:08:51.629804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.628610 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-local-certs" (OuterVolumeSpecName: "local-certs") pod "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" (UID: "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:08:51.629804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.628682 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-cacerts" (OuterVolumeSpecName: "cacerts") pod "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" (UID: "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:08:51.631230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.630076 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" (UID: "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:08:51.631230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.630188 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" (UID: "4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:08:51.663289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.663261 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv"] Sep 17 18:08:51.665479 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:08:51.665454 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod083acc7b_dc74_462f_b658_3a69fc43b013.slice/crio-d0f294b8dfca371417d69c0849545958ba4375ef6825a0f39e19af2c9ddbeabb WatchSource:0}: Error finding container d0f294b8dfca371417d69c0849545958ba4375ef6825a0f39e19af2c9ddbeabb: Status 404 returned error can't find the container with id d0f294b8dfca371417d69c0849545958ba4375ef6825a0f39e19af2c9ddbeabb Sep 17 18:08:51.726624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.726558 2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-kubeconfig\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:08:51.726624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.726585 2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-local-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:08:51.726624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.726593 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-dns-cert\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:08:51.726624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.726602 2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-token\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:08:51.726624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.726614 2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-cacerts\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:08:51.726910 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.726650 2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-istio-csr-ca-configmap\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:08:51.726910 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.726665 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8r8\" (UniqueName: \"kubernetes.io/projected/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf-kube-api-access-6k8r8\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:08:51.975591 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.975555 2582 generic.go:356] "Generic (PLEG): container finished" podID="4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" containerID="d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0" exitCode=0 Sep 17 18:08:51.976064 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.975620 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" Sep 17 18:08:51.976064 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.975720 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" event={"ID":"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf","Type":"ContainerDied","Data":"d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0"} Sep 17 18:08:51.976064 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.975757 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5" event={"ID":"4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf","Type":"ContainerDied","Data":"083faf37f4aed37f2737066eed38fab7ae24970af518bf3bc85260182f750da5"} Sep 17 18:08:51.976064 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.975780 2582 scope.go:117] "RemoveContainer" containerID="d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0" Sep 17 18:08:51.977538 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.977509 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" event={"ID":"083acc7b-dc74-462f-b658-3a69fc43b013","Type":"ContainerStarted","Data":"d0f294b8dfca371417d69c0849545958ba4375ef6825a0f39e19af2c9ddbeabb"} Sep 17 18:08:51.989013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.988966 2582 scope.go:117] "RemoveContainer" containerID="d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0" Sep 17 18:08:51.989395 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:08:51.989368 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0\": container with ID starting with d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0 not found: ID does not exist" containerID="d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0" Sep 17 18:08:51.989482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:51.989407 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0"} err="failed to get container status \"d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0\": rpc error: code = NotFound desc = could not find container \"d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0\": container with ID starting with d2fb028ef8f5c26d2dbbc6ea620e296e968795bdacf49a52b4c636769b5846c0 not found: ID does not exist" Sep 17 18:08:52.006050 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:52.006025 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5"] Sep 17 18:08:52.013137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:52.013112 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-57dw5"] Sep 17 18:08:53.350121 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:53.350045 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" path="/var/lib/kubelet/pods/4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf/volumes" Sep 17 18:08:53.992777 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:53.992701 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" event={"ID":"247b1774-3590-4aa0-b528-ab6f38a22d7a","Type":"ContainerStarted","Data":"154093708b8398568b8a30fd38821a4b83020be907151161ec43e85bda5dd648"} Sep 17 18:08:53.992978 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:53.992792 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:08:54.025520 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:54.025464 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" podStartSLOduration=2.139025011 podStartE2EDuration="5.025446136s" totalImagesPullingTime="2.886421125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:54.02294956 +0000 UTC m=+345.086893519" watchObservedRunningTime="2026-09-17 18:08:54.025446136 +0000 UTC m=+345.089390135" Sep 17 18:08:54.998632 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:54.998600 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" event={"ID":"083acc7b-dc74-462f-b658-3a69fc43b013","Type":"ContainerStarted","Data":"1baf21826c8910e84d285fdaf550e8e2b24b4b86c73fc8518d9d75277df25228"} Sep 17 18:08:54.999185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:08:54.998766 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:09:25.004155 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:25.004126 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bdzjt" Sep 17 18:09:25.023675 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:25.023625 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" podStartSLOduration=33.100248423 podStartE2EDuration="36.023609999s" totalImagesPullingTime="2.923361576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:55.038585782 +0000 UTC m=+346.102529739" watchObservedRunningTime="2026-09-17 18:09:25.023609999 +0000 UTC m=+376.087553956" Sep 17 18:09:26.005486 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:26.005458 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rftjv" Sep 17 18:09:46.241817 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.241780 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cfd75d48b-cbnf6"] Sep 17 18:09:46.242311 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.242222 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" containerName="discovery" Sep 17 18:09:46.242311 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.242238 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" containerName="discovery" Sep 17 18:09:46.242311 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.242301 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f04729e-1f1d-47ef-a0db-6f47ecc2b8cf" containerName="discovery" Sep 17 18:09:46.247241 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.247223 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.249841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.249819 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 17 18:09:46.250986 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.250971 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 17 18:09:46.251969 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.251951 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 17 18:09:46.253518 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.253498 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 17 18:09:46.253518 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.253498 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ppsr6\"" Sep 17 18:09:46.256914 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.256894 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 17 18:09:46.257019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.257004 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 17 18:09:46.266527 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.266503 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cfd75d48b-cbnf6"] Sep 17 18:09:46.289312 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.289276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13865664-45f9-45eb-a30c-341c56a07e50-console-serving-cert\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.289449 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.289329 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxlv6\" (UniqueName: \"kubernetes.io/projected/13865664-45f9-45eb-a30c-341c56a07e50-kube-api-access-lxlv6\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.289506 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.289481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-service-ca\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.289554 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.289519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-trusted-ca-bundle\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.289554 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.289541 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-console-config\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.289633 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.289555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-oauth-serving-cert\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.289633 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.289582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13865664-45f9-45eb-a30c-341c56a07e50-console-oauth-config\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.390801 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.390771 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-service-ca\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.390984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.390806 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-trusted-ca-bundle\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.390984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.390835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-console-config\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.390984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.390855 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-oauth-serving-cert\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.390984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.390905 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13865664-45f9-45eb-a30c-341c56a07e50-console-oauth-config\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.390984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.390957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13865664-45f9-45eb-a30c-341c56a07e50-console-serving-cert\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.391237 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.390993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxlv6\" (UniqueName: \"kubernetes.io/projected/13865664-45f9-45eb-a30c-341c56a07e50-kube-api-access-lxlv6\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.391637 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.391606 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-oauth-serving-cert\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.391751 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.391697 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-trusted-ca-bundle\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.391816 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.391769 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-console-config\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.391904 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.391823 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13865664-45f9-45eb-a30c-341c56a07e50-service-ca\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.393496 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.393471 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13865664-45f9-45eb-a30c-341c56a07e50-console-oauth-config\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.393590 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.393560 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13865664-45f9-45eb-a30c-341c56a07e50-console-serving-cert\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.402305 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.402283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxlv6\" (UniqueName: \"kubernetes.io/projected/13865664-45f9-45eb-a30c-341c56a07e50-kube-api-access-lxlv6\") pod \"console-5cfd75d48b-cbnf6\" (UID: \"13865664-45f9-45eb-a30c-341c56a07e50\") " pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.557431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.557354 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:46.836223 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:46.836197 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cfd75d48b-cbnf6"] Sep 17 18:09:46.838369 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:09:46.838343 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13865664_45f9_45eb_a30c_341c56a07e50.slice/crio-16378afff62828b4f7ccefb84d7b258569f810fe72b6dc5e29d39c92b4089427 WatchSource:0}: Error finding container 16378afff62828b4f7ccefb84d7b258569f810fe72b6dc5e29d39c92b4089427: Status 404 returned error can't find the container with id 16378afff62828b4f7ccefb84d7b258569f810fe72b6dc5e29d39c92b4089427 Sep 17 18:09:47.232410 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:47.232370 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfd75d48b-cbnf6" event={"ID":"13865664-45f9-45eb-a30c-341c56a07e50","Type":"ContainerStarted","Data":"edce5c8e4d44579d49981097392e296f35e9060ed3d2c8f5afaf537a601f5452"} Sep 17 18:09:47.232582 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:47.232453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cfd75d48b-cbnf6" event={"ID":"13865664-45f9-45eb-a30c-341c56a07e50","Type":"ContainerStarted","Data":"16378afff62828b4f7ccefb84d7b258569f810fe72b6dc5e29d39c92b4089427"} Sep 17 18:09:47.258555 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:47.258508 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cfd75d48b-cbnf6" podStartSLOduration=1.258493806 podStartE2EDuration="1.258493806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:47.256964539 +0000 UTC m=+398.320908498" watchObservedRunningTime="2026-09-17 18:09:47.258493806 +0000 UTC m=+398.322437804" Sep 17 18:09:56.557548 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:56.557506 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:56.558000 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:56.557570 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:56.562378 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:56.562357 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:57.287151 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.287114 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cfd75d48b-cbnf6" Sep 17 18:09:57.940990 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.940959 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j"] Sep 17 18:09:57.943350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.943335 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:09:57.946916 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.946890 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zkxj7\"" Sep 17 18:09:57.948131 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.948113 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 18:09:57.948209 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.948113 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 18:09:57.964485 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.964460 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j"] Sep 17 18:09:57.989812 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.989787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2nt\" (UniqueName: \"kubernetes.io/projected/0e18dc49-7a2c-4777-a292-f40470dbf6e5-kube-api-access-cg2nt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" (UID: \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:09:57.989951 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:57.989846 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e18dc49-7a2c-4777-a292-f40470dbf6e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" (UID: \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:09:58.091023 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:58.090989 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2nt\" (UniqueName: \"kubernetes.io/projected/0e18dc49-7a2c-4777-a292-f40470dbf6e5-kube-api-access-cg2nt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" (UID: \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:09:58.091157 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:58.091046 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e18dc49-7a2c-4777-a292-f40470dbf6e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" (UID: \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:09:58.091440 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:58.091421 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e18dc49-7a2c-4777-a292-f40470dbf6e5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" (UID: \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:09:58.104343 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:58.104316 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2nt\" (UniqueName: \"kubernetes.io/projected/0e18dc49-7a2c-4777-a292-f40470dbf6e5-kube-api-access-cg2nt\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" (UID: \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:09:58.253999 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:58.253918 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:09:58.578194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:58.578169 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j"] Sep 17 18:09:58.580998 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:09:58.580967 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e18dc49_7a2c_4777_a292_f40470dbf6e5.slice/crio-e9a7e65657e24ac5a94f9b91bd538221df49fa25543d6df7031840c4ac07c7e6 WatchSource:0}: Error finding container e9a7e65657e24ac5a94f9b91bd538221df49fa25543d6df7031840c4ac07c7e6: Status 404 returned error can't find the container with id e9a7e65657e24ac5a94f9b91bd538221df49fa25543d6df7031840c4ac07c7e6 Sep 17 18:09:59.295154 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:59.295116 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" event={"ID":"0e18dc49-7a2c-4777-a292-f40470dbf6e5","Type":"ContainerStarted","Data":"e9a7e65657e24ac5a94f9b91bd538221df49fa25543d6df7031840c4ac07c7e6"} Sep 17 18:09:59.862523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:59.862484 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6"] Sep 17 18:09:59.865823 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:59.865771 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:09:59.868720 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:59.868631 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5bkvp\"" Sep 17 18:09:59.881318 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:09:59.881289 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6"] Sep 17 18:10:00.009512 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:00.009477 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkpl\" (UniqueName: \"kubernetes.io/projected/c952a6c2-c09a-45ce-a938-8ab8af77a47f-kube-api-access-tfkpl\") pod \"limitador-operator-controller-manager-74495c69f8-l79l6\" (UID: \"c952a6c2-c09a-45ce-a938-8ab8af77a47f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:00.112165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:00.112097 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkpl\" (UniqueName: \"kubernetes.io/projected/c952a6c2-c09a-45ce-a938-8ab8af77a47f-kube-api-access-tfkpl\") pod \"limitador-operator-controller-manager-74495c69f8-l79l6\" (UID: \"c952a6c2-c09a-45ce-a938-8ab8af77a47f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:00.132719 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:00.132642 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkpl\" (UniqueName: \"kubernetes.io/projected/c952a6c2-c09a-45ce-a938-8ab8af77a47f-kube-api-access-tfkpl\") pod \"limitador-operator-controller-manager-74495c69f8-l79l6\" (UID: \"c952a6c2-c09a-45ce-a938-8ab8af77a47f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:00.181482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:00.181305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:00.503076 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:00.503045 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6"] Sep 17 18:10:01.223907 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:10:01.223852 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc952a6c2_c09a_45ce_a938_8ab8af77a47f.slice/crio-500519ace7c27ce61ff5b52c3dadfd3a1bc6b167392e72cdd8a21a2d07089aaf WatchSource:0}: Error finding container 500519ace7c27ce61ff5b52c3dadfd3a1bc6b167392e72cdd8a21a2d07089aaf: Status 404 returned error can't find the container with id 500519ace7c27ce61ff5b52c3dadfd3a1bc6b167392e72cdd8a21a2d07089aaf Sep 17 18:10:01.308752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:01.308716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" event={"ID":"c952a6c2-c09a-45ce-a938-8ab8af77a47f","Type":"ContainerStarted","Data":"500519ace7c27ce61ff5b52c3dadfd3a1bc6b167392e72cdd8a21a2d07089aaf"} Sep 17 18:10:02.633380 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:02.633342 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m4n7n"] Sep 17 18:10:02.636163 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:02.636141 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" Sep 17 18:10:02.638766 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:02.638745 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-r2g79\"" Sep 17 18:10:02.652787 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:02.652757 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m4n7n"] Sep 17 18:10:02.737872 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:02.737826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk6qm\" (UniqueName: \"kubernetes.io/projected/2fed7e09-62e3-420e-bd69-2fc1ae276c41-kube-api-access-gk6qm\") pod \"authorino-operator-76b478bd78-m4n7n\" (UID: \"2fed7e09-62e3-420e-bd69-2fc1ae276c41\") " pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" Sep 17 18:10:02.838893 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:02.838801 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk6qm\" (UniqueName: \"kubernetes.io/projected/2fed7e09-62e3-420e-bd69-2fc1ae276c41-kube-api-access-gk6qm\") pod \"authorino-operator-76b478bd78-m4n7n\" (UID: \"2fed7e09-62e3-420e-bd69-2fc1ae276c41\") " pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" Sep 17 18:10:02.863010 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:02.862985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk6qm\" (UniqueName: \"kubernetes.io/projected/2fed7e09-62e3-420e-bd69-2fc1ae276c41-kube-api-access-gk6qm\") pod \"authorino-operator-76b478bd78-m4n7n\" (UID: \"2fed7e09-62e3-420e-bd69-2fc1ae276c41\") " pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" Sep 17 18:10:02.948992 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:02.948970 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" Sep 17 18:10:03.227395 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:03.227372 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-m4n7n"] Sep 17 18:10:03.229029 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:10:03.229002 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fed7e09_62e3_420e_bd69_2fc1ae276c41.slice/crio-0177ee993a18617bbc8fde25552f21b248ce19cb98fc683a17aecd9049af3a41 WatchSource:0}: Error finding container 0177ee993a18617bbc8fde25552f21b248ce19cb98fc683a17aecd9049af3a41: Status 404 returned error can't find the container with id 0177ee993a18617bbc8fde25552f21b248ce19cb98fc683a17aecd9049af3a41 Sep 17 18:10:03.320382 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:03.320329 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" event={"ID":"0e18dc49-7a2c-4777-a292-f40470dbf6e5","Type":"ContainerStarted","Data":"6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568"} Sep 17 18:10:03.320556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:03.320399 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:03.321664 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:03.321632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" event={"ID":"2fed7e09-62e3-420e-bd69-2fc1ae276c41","Type":"ContainerStarted","Data":"0177ee993a18617bbc8fde25552f21b248ce19cb98fc683a17aecd9049af3a41"} Sep 17 18:10:03.350550 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:03.350503 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" podStartSLOduration=2.009609124 podStartE2EDuration="6.350488109s" totalImagesPullingTime="4.340878985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:03.347810487 +0000 UTC m=+414.411754444" watchObservedRunningTime="2026-09-17 18:10:03.350488109 +0000 UTC m=+414.414432067" Sep 17 18:10:04.327889 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:04.327844 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" event={"ID":"c952a6c2-c09a-45ce-a938-8ab8af77a47f","Type":"ContainerStarted","Data":"777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b"} Sep 17 18:10:04.366889 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:04.366823 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" podStartSLOduration=2.452037636 podStartE2EDuration="5.366807528s" totalImagesPullingTime="2.914769892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:04.364959405 +0000 UTC m=+415.428903374" watchObservedRunningTime="2026-09-17 18:10:04.366807528 +0000 UTC m=+415.430751486" Sep 17 18:10:05.090921 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.090857 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2"] Sep 17 18:10:05.093920 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.093895 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" Sep 17 18:10:05.096599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.096564 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 17 18:10:05.096738 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.096718 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9nh4s\"" Sep 17 18:10:05.118194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.118161 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2"] Sep 17 18:10:05.172181 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.172141 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmr9\" (UniqueName: \"kubernetes.io/projected/8b0de7a2-a141-408b-9385-9489b8903da2-kube-api-access-8hmr9\") pod \"dns-operator-controller-manager-7b84cd7c9c-2pjz2\" (UID: \"8b0de7a2-a141-408b-9385-9489b8903da2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" Sep 17 18:10:05.273449 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.273423 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmr9\" (UniqueName: \"kubernetes.io/projected/8b0de7a2-a141-408b-9385-9489b8903da2-kube-api-access-8hmr9\") pod \"dns-operator-controller-manager-7b84cd7c9c-2pjz2\" (UID: \"8b0de7a2-a141-408b-9385-9489b8903da2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" Sep 17 18:10:05.287292 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.287269 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmr9\" (UniqueName: \"kubernetes.io/projected/8b0de7a2-a141-408b-9385-9489b8903da2-kube-api-access-8hmr9\") pod \"dns-operator-controller-manager-7b84cd7c9c-2pjz2\" (UID: \"8b0de7a2-a141-408b-9385-9489b8903da2\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" Sep 17 18:10:05.338412 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.338357 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" event={"ID":"2fed7e09-62e3-420e-bd69-2fc1ae276c41","Type":"ContainerStarted","Data":"565952b32c3d469d57e0a046f85e834b6b57834da6ea92e2dba6cfa058cfb5a4"} Sep 17 18:10:05.338841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.338458 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" Sep 17 18:10:05.338841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.338517 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:05.378054 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.377950 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" podStartSLOduration=1.362989532 podStartE2EDuration="3.37793309s" totalImagesPullingTime="2.014943558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:05.376790273 +0000 UTC m=+416.440734248" watchObservedRunningTime="2026-09-17 18:10:05.37793309 +0000 UTC m=+416.441877047" Sep 17 18:10:05.408089 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.408060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" Sep 17 18:10:05.699190 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:05.699165 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2"] Sep 17 18:10:05.701840 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:10:05.701803 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0de7a2_a141_408b_9385_9489b8903da2.slice/crio-795b9d0749a8bf307c84567e90c3eaae02e0950319fd9bb2b1cd65a627ed1991 WatchSource:0}: Error finding container 795b9d0749a8bf307c84567e90c3eaae02e0950319fd9bb2b1cd65a627ed1991: Status 404 returned error can't find the container with id 795b9d0749a8bf307c84567e90c3eaae02e0950319fd9bb2b1cd65a627ed1991 Sep 17 18:10:06.344416 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:06.344381 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" event={"ID":"8b0de7a2-a141-408b-9385-9489b8903da2","Type":"ContainerStarted","Data":"795b9d0749a8bf307c84567e90c3eaae02e0950319fd9bb2b1cd65a627ed1991"} Sep 17 18:10:09.360194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:09.360157 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" event={"ID":"8b0de7a2-a141-408b-9385-9489b8903da2","Type":"ContainerStarted","Data":"cf075b17b9d6224ab9afea6376c89468d296655889cbfed29ba757e8b2b2b1fd"} Sep 17 18:10:09.360538 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:09.360271 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" Sep 17 18:10:09.383208 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:09.383159 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" podStartSLOduration=1.742156566 podStartE2EDuration="4.383144794s" totalImagesPullingTime="2.640988228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:09.381361385 +0000 UTC m=+420.445305342" watchObservedRunningTime="2026-09-17 18:10:09.383144794 +0000 UTC m=+420.447088752" Sep 17 18:10:14.330695 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:14.330662 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:16.346820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.346790 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:16.347323 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.346884 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-m4n7n" Sep 17 18:10:16.374022 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.373991 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9"] Sep 17 18:10:16.376622 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.376595 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:16.400098 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.400056 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9"] Sep 17 18:10:16.475681 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.475643 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dn6\" (UniqueName: \"kubernetes.io/projected/88155919-c253-42c0-b8d1-c71a564c5e56-kube-api-access-g9dn6\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" (UID: \"88155919-c253-42c0-b8d1-c71a564c5e56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:16.475883 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.475780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88155919-c253-42c0-b8d1-c71a564c5e56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" (UID: \"88155919-c253-42c0-b8d1-c71a564c5e56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:16.576641 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.576604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88155919-c253-42c0-b8d1-c71a564c5e56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" (UID: \"88155919-c253-42c0-b8d1-c71a564c5e56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:16.576841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.576758 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dn6\" (UniqueName: \"kubernetes.io/projected/88155919-c253-42c0-b8d1-c71a564c5e56-kube-api-access-g9dn6\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" (UID: \"88155919-c253-42c0-b8d1-c71a564c5e56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:16.577111 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.577087 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88155919-c253-42c0-b8d1-c71a564c5e56-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" (UID: \"88155919-c253-42c0-b8d1-c71a564c5e56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:16.595134 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.595102 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dn6\" (UniqueName: \"kubernetes.io/projected/88155919-c253-42c0-b8d1-c71a564c5e56-kube-api-access-g9dn6\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" (UID: \"88155919-c253-42c0-b8d1-c71a564c5e56\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:16.686623 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.686589 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:16.819858 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.819823 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j"] Sep 17 18:10:16.820054 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.820017 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" containerName="manager" containerID="cri-o://6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568" gracePeriod=2 Sep 17 18:10:16.839123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.839100 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j"] Sep 17 18:10:16.850706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.850681 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn"] Sep 17 18:10:16.851088 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.851074 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" containerName="manager" Sep 17 18:10:16.851088 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.851089 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" containerName="manager" Sep 17 18:10:16.851217 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.851156 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" containerName="manager" Sep 17 18:10:16.853421 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.853391 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:16.856038 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.856017 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9"] Sep 17 18:10:16.857406 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:16.857377 2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0e18dc49-7a2c-4777-a292-f40470dbf6e5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\": pods \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:16.861338 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:16.861312 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:16.866529 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.866506 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9"] Sep 17 18:10:16.878804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.878777 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn"] Sep 17 18:10:16.887261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.887232 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6"] Sep 17 18:10:16.887420 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.887396 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" containerName="manager" containerID="cri-o://777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b" gracePeriod=2 Sep 17 18:10:16.901030 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.901002 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn"] Sep 17 18:10:16.905843 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.905816 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:16.918923 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.918899 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6"] Sep 17 18:10:16.972476 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.969039 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn"] Sep 17 18:10:16.972476 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:16.972175 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:16.975306 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.975274 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj"] Sep 17 18:10:16.976158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.976036 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" containerName="manager" Sep 17 18:10:16.976158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.976062 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" containerName="manager" Sep 17 18:10:16.976342 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.976204 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" containerName="manager" Sep 17 18:10:16.979707 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.979685 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" Sep 17 18:10:16.979995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.979970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5gv\" (UniqueName: \"kubernetes.io/projected/91b93787-94f3-4035-a41f-c04341c422b8-kube-api-access-hb5gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvmmn\" (UID: \"91b93787-94f3-4035-a41f-c04341c422b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:16.980184 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.980167 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91b93787-94f3-4035-a41f-c04341c422b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvmmn\" (UID: \"91b93787-94f3-4035-a41f-c04341c422b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:16.982338 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:16.982315 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj"] Sep 17 18:10:17.014788 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.014696 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l79l6\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:17.017377 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.017311 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:17.042950 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.042892 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l79l6\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:17.083200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.082239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5gv\" (UniqueName: \"kubernetes.io/projected/91b93787-94f3-4035-a41f-c04341c422b8-kube-api-access-hb5gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvmmn\" (UID: \"91b93787-94f3-4035-a41f-c04341c422b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:17.083200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.082334 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91b93787-94f3-4035-a41f-c04341c422b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvmmn\" (UID: \"91b93787-94f3-4035-a41f-c04341c422b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:17.083200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.082396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwhs\" (UniqueName: \"kubernetes.io/projected/d303f06f-ac48-49ec-b022-ae2d4a4905da-kube-api-access-sxwhs\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s28pn\" (UID: \"d303f06f-ac48-49ec-b022-ae2d4a4905da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:17.083200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.082430 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdb92\" (UniqueName: \"kubernetes.io/projected/13af60a1-2f0c-4ff4-8393-83169bcabd09-kube-api-access-xdb92\") pod \"limitador-operator-controller-manager-74495c69f8-g59qj\" (UID: \"13af60a1-2f0c-4ff4-8393-83169bcabd09\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" Sep 17 18:10:17.083200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.082491 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d303f06f-ac48-49ec-b022-ae2d4a4905da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s28pn\" (UID: \"d303f06f-ac48-49ec-b022-ae2d4a4905da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:17.083573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.083258 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91b93787-94f3-4035-a41f-c04341c422b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvmmn\" (UID: \"91b93787-94f3-4035-a41f-c04341c422b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:17.095055 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.095027 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5gv\" (UniqueName: \"kubernetes.io/projected/91b93787-94f3-4035-a41f-c04341c422b8-kube-api-access-hb5gv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-dvmmn\" (UID: \"91b93787-94f3-4035-a41f-c04341c422b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:17.125327 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.125293 2582 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 17 18:10:17.125327 ip-10-0-137-104 kubenswrapper[2582]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-c9pw9_kuadrant-system_88155919-c253-42c0-b8d1-c71a564c5e56_0(7f0034fedf79d7979c94fc3990f2432d17ce45fa06b9c079ace105a6f1a4300b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-c9pw9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7f0034fedf79d7979c94fc3990f2432d17ce45fa06b9c079ace105a6f1a4300b" Netns:"/var/run/netns/777b5382-c246-41cd-a184-70d067998830" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-c9pw9;K8S_POD_INFRA_CONTAINER_ID=7f0034fedf79d7979c94fc3990f2432d17ce45fa06b9c079ace105a6f1a4300b;K8S_POD_UID=88155919-c253-42c0-b8d1-c71a564c5e56" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9/88155919-c253-42c0-b8d1-c71a564c5e56:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9 7f0034fedf79d7979c94fc3990f2432d17ce45fa06b9c079ace105a6f1a4300b network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:24 [10.134.0.36/23] Sep 17 18:10:17.125327 ip-10-0-137-104 kubenswrapper[2582]: ' Sep 17 18:10:17.125327 ip-10-0-137-104 kubenswrapper[2582]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 17 18:10:17.125327 ip-10-0-137-104 kubenswrapper[2582]: > Sep 17 18:10:17.125518 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.125355 2582 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 17 18:10:17.125518 ip-10-0-137-104 kubenswrapper[2582]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-c9pw9_kuadrant-system_88155919-c253-42c0-b8d1-c71a564c5e56_0(7f0034fedf79d7979c94fc3990f2432d17ce45fa06b9c079ace105a6f1a4300b): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-c9pw9 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"7f0034fedf79d7979c94fc3990f2432d17ce45fa06b9c079ace105a6f1a4300b" Netns:"/var/run/netns/777b5382-c246-41cd-a184-70d067998830" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-c9pw9;K8S_POD_INFRA_CONTAINER_ID=7f0034fedf79d7979c94fc3990f2432d17ce45fa06b9c079ace105a6f1a4300b;K8S_POD_UID=88155919-c253-42c0-b8d1-c71a564c5e56" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9/88155919-c253-42c0-b8d1-c71a564c5e56:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9 7f0034fedf79d7979c94fc3990f2432d17ce45fa06b9c079ace105a6f1a4300b network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:24 [10.134.0.36/23] Sep 17 18:10:17.125518 ip-10-0-137-104 kubenswrapper[2582]: ' Sep 17 18:10:17.125518 ip-10-0-137-104 kubenswrapper[2582]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 17 18:10:17.125518 ip-10-0-137-104 kubenswrapper[2582]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:17.129475 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.129450 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:17.132672 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.132639 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l79l6\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:17.134684 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.134659 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:17.136930 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.136914 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:17.139062 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.139037 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-l79l6\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:17.141451 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.141427 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fnx5j\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:17.183274 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.183240 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwhs\" (UniqueName: \"kubernetes.io/projected/d303f06f-ac48-49ec-b022-ae2d4a4905da-kube-api-access-sxwhs\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s28pn\" (UID: \"d303f06f-ac48-49ec-b022-ae2d4a4905da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:17.183274 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.183277 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdb92\" (UniqueName: \"kubernetes.io/projected/13af60a1-2f0c-4ff4-8393-83169bcabd09-kube-api-access-xdb92\") pod \"limitador-operator-controller-manager-74495c69f8-g59qj\" (UID: \"13af60a1-2f0c-4ff4-8393-83169bcabd09\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" Sep 17 18:10:17.183469 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.183320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d303f06f-ac48-49ec-b022-ae2d4a4905da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s28pn\" (UID: \"d303f06f-ac48-49ec-b022-ae2d4a4905da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:17.183721 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.183700 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d303f06f-ac48-49ec-b022-ae2d4a4905da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s28pn\" (UID: \"d303f06f-ac48-49ec-b022-ae2d4a4905da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:17.206299 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.206234 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdb92\" (UniqueName: \"kubernetes.io/projected/13af60a1-2f0c-4ff4-8393-83169bcabd09-kube-api-access-xdb92\") pod \"limitador-operator-controller-manager-74495c69f8-g59qj\" (UID: \"13af60a1-2f0c-4ff4-8393-83169bcabd09\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" Sep 17 18:10:17.206299 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.206248 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwhs\" (UniqueName: \"kubernetes.io/projected/d303f06f-ac48-49ec-b022-ae2d4a4905da-kube-api-access-sxwhs\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-s28pn\" (UID: \"d303f06f-ac48-49ec-b022-ae2d4a4905da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:17.273915 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.273854 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:17.283982 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.283955 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfkpl\" (UniqueName: \"kubernetes.io/projected/c952a6c2-c09a-45ce-a938-8ab8af77a47f-kube-api-access-tfkpl\") pod \"c952a6c2-c09a-45ce-a938-8ab8af77a47f\" (UID: \"c952a6c2-c09a-45ce-a938-8ab8af77a47f\") " Sep 17 18:10:17.284136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.284112 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e18dc49-7a2c-4777-a292-f40470dbf6e5-extensions-socket-volume\") pod \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\" (UID: \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\") " Sep 17 18:10:17.284272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.284152 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg2nt\" (UniqueName: \"kubernetes.io/projected/0e18dc49-7a2c-4777-a292-f40470dbf6e5-kube-api-access-cg2nt\") pod \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\" (UID: \"0e18dc49-7a2c-4777-a292-f40470dbf6e5\") " Sep 17 18:10:17.284272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.284212 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:17.284552 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.284529 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e18dc49-7a2c-4777-a292-f40470dbf6e5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0e18dc49-7a2c-4777-a292-f40470dbf6e5" (UID: "0e18dc49-7a2c-4777-a292-f40470dbf6e5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:10:17.286368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.286325 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c952a6c2-c09a-45ce-a938-8ab8af77a47f-kube-api-access-tfkpl" (OuterVolumeSpecName: "kube-api-access-tfkpl") pod "c952a6c2-c09a-45ce-a938-8ab8af77a47f" (UID: "c952a6c2-c09a-45ce-a938-8ab8af77a47f"). InnerVolumeSpecName "kube-api-access-tfkpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:10:17.286467 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.286378 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e18dc49-7a2c-4777-a292-f40470dbf6e5-kube-api-access-cg2nt" (OuterVolumeSpecName: "kube-api-access-cg2nt") pod "0e18dc49-7a2c-4777-a292-f40470dbf6e5" (UID: "0e18dc49-7a2c-4777-a292-f40470dbf6e5"). InnerVolumeSpecName "kube-api-access-cg2nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:10:17.295239 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.295214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" Sep 17 18:10:17.352322 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.350549 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" path="/var/lib/kubelet/pods/0e18dc49-7a2c-4777-a292-f40470dbf6e5/volumes" Sep 17 18:10:17.352322 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.351105 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" path="/var/lib/kubelet/pods/c952a6c2-c09a-45ce-a938-8ab8af77a47f/volumes" Sep 17 18:10:17.385608 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.385582 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e18dc49-7a2c-4777-a292-f40470dbf6e5-extensions-socket-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:17.385608 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.385607 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg2nt\" (UniqueName: \"kubernetes.io/projected/0e18dc49-7a2c-4777-a292-f40470dbf6e5-kube-api-access-cg2nt\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:17.385761 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.385619 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfkpl\" (UniqueName: \"kubernetes.io/projected/c952a6c2-c09a-45ce-a938-8ab8af77a47f-kube-api-access-tfkpl\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:17.406344 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.406314 2582 generic.go:356] "Generic (PLEG): container finished" podID="0e18dc49-7a2c-4777-a292-f40470dbf6e5" containerID="6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568" exitCode=0 Sep 17 18:10:17.406504 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.406379 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fnx5j" Sep 17 18:10:17.406504 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.406417 2582 scope.go:117] "RemoveContainer" containerID="6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568" Sep 17 18:10:17.410344 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.408434 2582 generic.go:356] "Generic (PLEG): container finished" podID="c952a6c2-c09a-45ce-a938-8ab8af77a47f" containerID="777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b" exitCode=0 Sep 17 18:10:17.410344 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.408507 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:17.410525 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.408992 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-l79l6" Sep 17 18:10:17.418543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.418523 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:17.418620 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.418546 2582 scope.go:117] "RemoveContainer" containerID="6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568" Sep 17 18:10:17.418854 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.418832 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568\": container with ID starting with 6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568 not found: ID does not exist" containerID="6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568" Sep 17 18:10:17.418999 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.418859 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568"} err="failed to get container status \"6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568\": rpc error: code = NotFound desc = could not find container \"6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568\": container with ID starting with 6aac0cceeaed55b004be04adb6aa67f4265ceab43d0abdecc7cd2d195efa4568 not found: ID does not exist" Sep 17 18:10:17.418999 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.418899 2582 scope.go:117] "RemoveContainer" containerID="777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b" Sep 17 18:10:17.421039 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.421013 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="88155919-c253-42c0-b8d1-c71a564c5e56" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:17.426816 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.426796 2582 scope.go:117] "RemoveContainer" containerID="777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b" Sep 17 18:10:17.427163 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:17.427142 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b\": container with ID starting with 777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b not found: ID does not exist" containerID="777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b" Sep 17 18:10:17.427254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.427171 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b"} err="failed to get container status \"777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b\": rpc error: code = NotFound desc = could not find container \"777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b\": container with ID starting with 777dbd63466dc15c7de997266d7b97c579c6e12c2e39c3a3caaf63b2fdb0780b not found: ID does not exist" Sep 17 18:10:17.569387 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.569308 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn"] Sep 17 18:10:17.572960 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:10:17.572917 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b93787_94f3_4035_a41f_c04341c422b8.slice/crio-144c9d42b82090ab861f1c1a2b98dacc67e26a3c4bca115bf7943f591cba8187 WatchSource:0}: Error finding container 144c9d42b82090ab861f1c1a2b98dacc67e26a3c4bca115bf7943f591cba8187: Status 404 returned error can't find the container with id 144c9d42b82090ab861f1c1a2b98dacc67e26a3c4bca115bf7943f591cba8187 Sep 17 18:10:17.587230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.587203 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9dn6\" (UniqueName: \"kubernetes.io/projected/88155919-c253-42c0-b8d1-c71a564c5e56-kube-api-access-g9dn6\") pod \"88155919-c253-42c0-b8d1-c71a564c5e56\" (UID: \"88155919-c253-42c0-b8d1-c71a564c5e56\") " Sep 17 18:10:17.587346 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.587284 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88155919-c253-42c0-b8d1-c71a564c5e56-extensions-socket-volume\") pod \"88155919-c253-42c0-b8d1-c71a564c5e56\" (UID: \"88155919-c253-42c0-b8d1-c71a564c5e56\") " Sep 17 18:10:17.587517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.587496 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88155919-c253-42c0-b8d1-c71a564c5e56-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "88155919-c253-42c0-b8d1-c71a564c5e56" (UID: "88155919-c253-42c0-b8d1-c71a564c5e56"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:10:17.587635 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.587620 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88155919-c253-42c0-b8d1-c71a564c5e56-extensions-socket-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:17.589375 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.589348 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88155919-c253-42c0-b8d1-c71a564c5e56-kube-api-access-g9dn6" (OuterVolumeSpecName: "kube-api-access-g9dn6") pod "88155919-c253-42c0-b8d1-c71a564c5e56" (UID: "88155919-c253-42c0-b8d1-c71a564c5e56"). InnerVolumeSpecName "kube-api-access-g9dn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:10:17.594620 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.594598 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn"] Sep 17 18:10:17.599568 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.599547 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj"] Sep 17 18:10:17.603399 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:10:17.603370 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd303f06f_ac48_49ec_b022_ae2d4a4905da.slice/crio-23c4061eec3763241fec948c7b4ecc2c02e50411b51ec23b54185010e0a64fec WatchSource:0}: Error finding container 23c4061eec3763241fec948c7b4ecc2c02e50411b51ec23b54185010e0a64fec: Status 404 returned error can't find the container with id 23c4061eec3763241fec948c7b4ecc2c02e50411b51ec23b54185010e0a64fec Sep 17 18:10:17.603646 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:10:17.603625 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13af60a1_2f0c_4ff4_8393_83169bcabd09.slice/crio-edebed7da9e99674858cf0facee05a46fa829fcd12f8069a3e476d65e8236108 WatchSource:0}: Error finding container edebed7da9e99674858cf0facee05a46fa829fcd12f8069a3e476d65e8236108: Status 404 returned error can't find the container with id edebed7da9e99674858cf0facee05a46fa829fcd12f8069a3e476d65e8236108 Sep 17 18:10:17.688737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:17.688709 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dn6\" (UniqueName: \"kubernetes.io/projected/88155919-c253-42c0-b8d1-c71a564c5e56-kube-api-access-g9dn6\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:18.416202 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.416165 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" event={"ID":"d303f06f-ac48-49ec-b022-ae2d4a4905da","Type":"ContainerStarted","Data":"1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7"} Sep 17 18:10:18.416691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.416233 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:18.416691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.416258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" event={"ID":"d303f06f-ac48-49ec-b022-ae2d4a4905da","Type":"ContainerStarted","Data":"23c4061eec3763241fec948c7b4ecc2c02e50411b51ec23b54185010e0a64fec"} Sep 17 18:10:18.418473 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:18.418438 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="88155919-c253-42c0-b8d1-c71a564c5e56" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:18.421463 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.421431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" event={"ID":"91b93787-94f3-4035-a41f-c04341c422b8","Type":"ContainerStarted","Data":"bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569"} Sep 17 18:10:18.421612 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.421487 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:18.421612 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.421505 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" event={"ID":"91b93787-94f3-4035-a41f-c04341c422b8","Type":"ContainerStarted","Data":"144c9d42b82090ab861f1c1a2b98dacc67e26a3c4bca115bf7943f591cba8187"} Sep 17 18:10:18.423799 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.423778 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:18.423918 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.423793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" event={"ID":"13af60a1-2f0c-4ff4-8393-83169bcabd09","Type":"ContainerStarted","Data":"fe6fec60517a90d9a74b29d033fa4fcc62347d610e998e463d9ea7106591b145"} Sep 17 18:10:18.423918 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.423833 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" event={"ID":"13af60a1-2f0c-4ff4-8393-83169bcabd09","Type":"ContainerStarted","Data":"edebed7da9e99674858cf0facee05a46fa829fcd12f8069a3e476d65e8236108"} Sep 17 18:10:18.423918 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.423910 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" Sep 17 18:10:18.455953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.455909 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" podStartSLOduration=2.455893144 podStartE2EDuration="2.455893144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:18.453196336 +0000 UTC m=+429.517140297" watchObservedRunningTime="2026-09-17 18:10:18.455893144 +0000 UTC m=+429.519837092" Sep 17 18:10:18.456086 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:18.456019 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-c9pw9\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="88155919-c253-42c0-b8d1-c71a564c5e56" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-c9pw9" Sep 17 18:10:18.479296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.479250 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" podStartSLOduration=2.47923489 podStartE2EDuration="2.47923489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:18.477260686 +0000 UTC m=+429.541204645" watchObservedRunningTime="2026-09-17 18:10:18.47923489 +0000 UTC m=+429.543178847" Sep 17 18:10:18.502161 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:18.502121 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" podStartSLOduration=2.502104986 podStartE2EDuration="2.502104986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:18.500122417 +0000 UTC m=+429.564066375" watchObservedRunningTime="2026-09-17 18:10:18.502104986 +0000 UTC m=+429.566048944" Sep 17 18:10:19.349985 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:19.347205 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88155919-c253-42c0-b8d1-c71a564c5e56" path="/var/lib/kubelet/pods/88155919-c253-42c0-b8d1-c71a564c5e56/volumes" Sep 17 18:10:20.366958 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:20.366930 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-2pjz2" Sep 17 18:10:29.432448 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.432352 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:29.432448 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.432438 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:29.433045 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.432619 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g59qj" Sep 17 18:10:29.528312 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.528275 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn"] Sep 17 18:10:29.528559 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.528534 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" podUID="91b93787-94f3-4035-a41f-c04341c422b8" containerName="manager" containerID="cri-o://bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569" gracePeriod=10 Sep 17 18:10:29.800382 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.800352 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk"] Sep 17 18:10:29.807210 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.807188 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:29.813029 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.813010 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:29.832977 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.832950 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk"] Sep 17 18:10:29.884603 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.884574 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb5gv\" (UniqueName: \"kubernetes.io/projected/91b93787-94f3-4035-a41f-c04341c422b8-kube-api-access-hb5gv\") pod \"91b93787-94f3-4035-a41f-c04341c422b8\" (UID: \"91b93787-94f3-4035-a41f-c04341c422b8\") " Sep 17 18:10:29.884777 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.884632 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91b93787-94f3-4035-a41f-c04341c422b8-extensions-socket-volume\") pod \"91b93787-94f3-4035-a41f-c04341c422b8\" (UID: \"91b93787-94f3-4035-a41f-c04341c422b8\") " Sep 17 18:10:29.884828 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.884773 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w2frk\" (UID: \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:29.884903 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.884831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tcth\" (UniqueName: \"kubernetes.io/projected/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-kube-api-access-5tcth\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w2frk\" (UID: \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:29.885053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.885023 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b93787-94f3-4035-a41f-c04341c422b8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "91b93787-94f3-4035-a41f-c04341c422b8" (UID: "91b93787-94f3-4035-a41f-c04341c422b8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:10:29.886634 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.886614 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b93787-94f3-4035-a41f-c04341c422b8-kube-api-access-hb5gv" (OuterVolumeSpecName: "kube-api-access-hb5gv") pod "91b93787-94f3-4035-a41f-c04341c422b8" (UID: "91b93787-94f3-4035-a41f-c04341c422b8"). InnerVolumeSpecName "kube-api-access-hb5gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:10:29.985949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.985833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w2frk\" (UID: \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:29.985949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.985918 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tcth\" (UniqueName: \"kubernetes.io/projected/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-kube-api-access-5tcth\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w2frk\" (UID: \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:29.986179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.985988 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5gv\" (UniqueName: \"kubernetes.io/projected/91b93787-94f3-4035-a41f-c04341c422b8-kube-api-access-hb5gv\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:29.986179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.985998 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/91b93787-94f3-4035-a41f-c04341c422b8-extensions-socket-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:29.986293 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:29.986191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w2frk\" (UID: \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:30.000805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.000782 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tcth\" (UniqueName: \"kubernetes.io/projected/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-kube-api-access-5tcth\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w2frk\" (UID: \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:30.122546 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.122512 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:30.429244 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.429163 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk"] Sep 17 18:10:30.432186 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:10:30.432162 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afe7ab4_a326_4fb6_b9b5_c894e59c317b.slice/crio-83ff4f7220cf18a2eaa4f7b9808b968a718eff812c0bee91d3185dec4d056aff WatchSource:0}: Error finding container 83ff4f7220cf18a2eaa4f7b9808b968a718eff812c0bee91d3185dec4d056aff: Status 404 returned error can't find the container with id 83ff4f7220cf18a2eaa4f7b9808b968a718eff812c0bee91d3185dec4d056aff Sep 17 18:10:30.493052 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.493023 2582 generic.go:356] "Generic (PLEG): container finished" podID="91b93787-94f3-4035-a41f-c04341c422b8" containerID="bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569" exitCode=0 Sep 17 18:10:30.493466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.493090 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" Sep 17 18:10:30.493466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.493108 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" event={"ID":"91b93787-94f3-4035-a41f-c04341c422b8","Type":"ContainerDied","Data":"bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569"} Sep 17 18:10:30.493466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.493167 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn" event={"ID":"91b93787-94f3-4035-a41f-c04341c422b8","Type":"ContainerDied","Data":"144c9d42b82090ab861f1c1a2b98dacc67e26a3c4bca115bf7943f591cba8187"} Sep 17 18:10:30.493466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.493201 2582 scope.go:117] "RemoveContainer" containerID="bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569" Sep 17 18:10:30.495380 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.495357 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" event={"ID":"8afe7ab4-a326-4fb6-b9b5-c894e59c317b","Type":"ContainerStarted","Data":"83ff4f7220cf18a2eaa4f7b9808b968a718eff812c0bee91d3185dec4d056aff"} Sep 17 18:10:30.502751 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.502732 2582 scope.go:117] "RemoveContainer" containerID="bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569" Sep 17 18:10:30.503058 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:30.503027 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569\": container with ID starting with bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569 not found: ID does not exist" containerID="bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569" Sep 17 18:10:30.503123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.503059 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569"} err="failed to get container status \"bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569\": rpc error: code = NotFound desc = could not find container \"bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569\": container with ID starting with bfcab4eecc3942299e69099ee0f59f7e5e152f1d7dbf94007a2c55d43464b569 not found: ID does not exist" Sep 17 18:10:30.530545 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.530520 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn"] Sep 17 18:10:30.544372 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:30.544348 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-dvmmn"] Sep 17 18:10:31.346232 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:31.346199 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91b93787-94f3-4035-a41f-c04341c422b8" path="/var/lib/kubelet/pods/91b93787-94f3-4035-a41f-c04341c422b8/volumes" Sep 17 18:10:31.503342 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:31.503309 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" event={"ID":"8afe7ab4-a326-4fb6-b9b5-c894e59c317b","Type":"ContainerStarted","Data":"fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683"} Sep 17 18:10:31.503740 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:31.503375 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:31.528753 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:31.528707 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" podStartSLOduration=2.528690413 podStartE2EDuration="2.528690413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:31.526828738 +0000 UTC m=+442.590772696" watchObservedRunningTime="2026-09-17 18:10:31.528690413 +0000 UTC m=+442.592634370" Sep 17 18:10:42.512732 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:42.512701 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:10:42.575297 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:42.575264 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn"] Sep 17 18:10:42.575568 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:42.575520 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" podUID="d303f06f-ac48-49ec-b022-ae2d4a4905da" containerName="manager" containerID="cri-o://1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7" gracePeriod=10 Sep 17 18:10:42.819930 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:42.819906 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:42.899837 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:42.899805 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d303f06f-ac48-49ec-b022-ae2d4a4905da-extensions-socket-volume\") pod \"d303f06f-ac48-49ec-b022-ae2d4a4905da\" (UID: \"d303f06f-ac48-49ec-b022-ae2d4a4905da\") " Sep 17 18:10:42.900019 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:42.899881 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxwhs\" (UniqueName: \"kubernetes.io/projected/d303f06f-ac48-49ec-b022-ae2d4a4905da-kube-api-access-sxwhs\") pod \"d303f06f-ac48-49ec-b022-ae2d4a4905da\" (UID: \"d303f06f-ac48-49ec-b022-ae2d4a4905da\") " Sep 17 18:10:42.900269 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:42.900244 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d303f06f-ac48-49ec-b022-ae2d4a4905da-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d303f06f-ac48-49ec-b022-ae2d4a4905da" (UID: "d303f06f-ac48-49ec-b022-ae2d4a4905da"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:10:42.901826 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:42.901804 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d303f06f-ac48-49ec-b022-ae2d4a4905da-kube-api-access-sxwhs" (OuterVolumeSpecName: "kube-api-access-sxwhs") pod "d303f06f-ac48-49ec-b022-ae2d4a4905da" (UID: "d303f06f-ac48-49ec-b022-ae2d4a4905da"). InnerVolumeSpecName "kube-api-access-sxwhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:10:43.000877 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.000842 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxwhs\" (UniqueName: \"kubernetes.io/projected/d303f06f-ac48-49ec-b022-ae2d4a4905da-kube-api-access-sxwhs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:43.000877 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.000882 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d303f06f-ac48-49ec-b022-ae2d4a4905da-extensions-socket-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:10:43.579361 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.579327 2582 generic.go:356] "Generic (PLEG): container finished" podID="d303f06f-ac48-49ec-b022-ae2d4a4905da" containerID="1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7" exitCode=0 Sep 17 18:10:43.579764 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.579367 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" Sep 17 18:10:43.579764 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.579374 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" event={"ID":"d303f06f-ac48-49ec-b022-ae2d4a4905da","Type":"ContainerDied","Data":"1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7"} Sep 17 18:10:43.579764 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.579418 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn" event={"ID":"d303f06f-ac48-49ec-b022-ae2d4a4905da","Type":"ContainerDied","Data":"23c4061eec3763241fec948c7b4ecc2c02e50411b51ec23b54185010e0a64fec"} Sep 17 18:10:43.579764 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.579440 2582 scope.go:117] "RemoveContainer" containerID="1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7" Sep 17 18:10:43.588205 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.588187 2582 scope.go:117] "RemoveContainer" containerID="1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7" Sep 17 18:10:43.588459 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:10:43.588440 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7\": container with ID starting with 1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7 not found: ID does not exist" containerID="1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7" Sep 17 18:10:43.588518 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.588466 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7"} err="failed to get container status \"1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7\": rpc error: code = NotFound desc = could not find container \"1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7\": container with ID starting with 1f31391163804e7c3b2884b525548bd552494d40d9ef73e5aee7f3a6a3f76db7 not found: ID does not exist" Sep 17 18:10:43.603590 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.603531 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn"] Sep 17 18:10:43.614797 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:43.614778 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-s28pn"] Sep 17 18:10:45.345748 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.345704 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d303f06f-ac48-49ec-b022-ae2d4a4905da" path="/var/lib/kubelet/pods/d303f06f-ac48-49ec-b022-ae2d4a4905da/volumes" Sep 17 18:10:45.956956 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.956918 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt"] Sep 17 18:10:45.957352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.957335 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d303f06f-ac48-49ec-b022-ae2d4a4905da" containerName="manager" Sep 17 18:10:45.957453 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.957354 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d303f06f-ac48-49ec-b022-ae2d4a4905da" containerName="manager" Sep 17 18:10:45.957453 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.957402 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91b93787-94f3-4035-a41f-c04341c422b8" containerName="manager" Sep 17 18:10:45.957453 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.957411 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="91b93787-94f3-4035-a41f-c04341c422b8" containerName="manager" Sep 17 18:10:45.957598 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.957500 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d303f06f-ac48-49ec-b022-ae2d4a4905da" containerName="manager" Sep 17 18:10:45.957598 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.957516 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="91b93787-94f3-4035-a41f-c04341c422b8" containerName="manager" Sep 17 18:10:45.960103 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.960082 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:45.962703 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.962682 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 17 18:10:45.962805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.962682 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lmvxh\"" Sep 17 18:10:45.973124 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:45.973101 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt"] Sep 17 18:10:46.026356 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.026356 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.026573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.026573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.026573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026554 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.026697 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.026697 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f850692-e665-4012-bc51-ab3bd9c42ab0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.026697 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbcp7\" (UniqueName: \"kubernetes.io/projected/0f850692-e665-4012-bc51-ab3bd9c42ab0-kube-api-access-rbcp7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.026697 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.026675 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128031 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.127996 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128043 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128096 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f850692-e665-4012-bc51-ab3bd9c42ab0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbcp7\" (UniqueName: \"kubernetes.io/projected/0f850692-e665-4012-bc51-ab3bd9c42ab0-kube-api-access-rbcp7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128185 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128179 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128244 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128640 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128612 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128761 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128704 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128761 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128750 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.128947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.128930 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f850692-e665-4012-bc51-ab3bd9c42ab0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.130328 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.130307 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.130618 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.130600 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.136956 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.136935 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbcp7\" (UniqueName: \"kubernetes.io/projected/0f850692-e665-4012-bc51-ab3bd9c42ab0-kube-api-access-rbcp7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.137049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.137026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f850692-e665-4012-bc51-ab3bd9c42ab0-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-6v2jt\" (UID: \"0f850692-e665-4012-bc51-ab3bd9c42ab0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.270639 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.270567 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:46.545444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.545422 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt"] Sep 17 18:10:46.547438 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:10:46.547408 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f850692_e665_4012_bc51_ab3bd9c42ab0.slice/crio-00095d8f0b52463514beb6ab0d2636c6e5c6c81a67867f02a0de3446343064fe WatchSource:0}: Error finding container 00095d8f0b52463514beb6ab0d2636c6e5c6c81a67867f02a0de3446343064fe: Status 404 returned error can't find the container with id 00095d8f0b52463514beb6ab0d2636c6e5c6c81a67867f02a0de3446343064fe Sep 17 18:10:46.598640 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:46.598608 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" event={"ID":"0f850692-e665-4012-bc51-ab3bd9c42ab0","Type":"ContainerStarted","Data":"00095d8f0b52463514beb6ab0d2636c6e5c6c81a67867f02a0de3446343064fe"} Sep 17 18:10:49.435710 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:49.435669 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 18:10:49.436016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:49.435744 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 18:10:49.436016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:49.435775 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 18:10:49.620765 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:49.620729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" event={"ID":"0f850692-e665-4012-bc51-ab3bd9c42ab0","Type":"ContainerStarted","Data":"918a325152081573d6017b4a7818cbd6c0ae66fccc291143f8d422f0a8043656"} Sep 17 18:10:49.643664 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:49.643607 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" podStartSLOduration=1.75774118 podStartE2EDuration="4.643591668s" totalImagesPullingTime="2.885850488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:49.640951854 +0000 UTC m=+460.704895813" watchObservedRunningTime="2026-09-17 18:10:49.643591668 +0000 UTC m=+460.707535626" Sep 17 18:10:50.271196 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:50.271166 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:50.275981 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:50.275957 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:50.625602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:50.625509 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:10:50.626408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:10:50.626387 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:11:00.490598 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.490563 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xrhbn"] Sep 17 18:11:00.494125 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.494096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:00.498549 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.498524 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 17 18:11:00.498693 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.498545 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vb8rw\"" Sep 17 18:11:00.503914 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.503892 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xrhbn"] Sep 17 18:11:00.587555 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.587525 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xrhbn"] Sep 17 18:11:00.661828 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.661798 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tlwm\" (UniqueName: \"kubernetes.io/projected/f8cb3948-49b7-46a5-b2df-92239726c45f-kube-api-access-8tlwm\") pod \"limitador-limitador-8588749977-xrhbn\" (UID: \"f8cb3948-49b7-46a5-b2df-92239726c45f\") " pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:00.662014 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.661848 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8cb3948-49b7-46a5-b2df-92239726c45f-config-file\") pod \"limitador-limitador-8588749977-xrhbn\" (UID: \"f8cb3948-49b7-46a5-b2df-92239726c45f\") " pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:00.762484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.762404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tlwm\" (UniqueName: \"kubernetes.io/projected/f8cb3948-49b7-46a5-b2df-92239726c45f-kube-api-access-8tlwm\") pod \"limitador-limitador-8588749977-xrhbn\" (UID: \"f8cb3948-49b7-46a5-b2df-92239726c45f\") " pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:00.762484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.762446 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8cb3948-49b7-46a5-b2df-92239726c45f-config-file\") pod \"limitador-limitador-8588749977-xrhbn\" (UID: \"f8cb3948-49b7-46a5-b2df-92239726c45f\") " pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:00.763057 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.763040 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8cb3948-49b7-46a5-b2df-92239726c45f-config-file\") pod \"limitador-limitador-8588749977-xrhbn\" (UID: \"f8cb3948-49b7-46a5-b2df-92239726c45f\") " pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:00.770852 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.770825 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tlwm\" (UniqueName: \"kubernetes.io/projected/f8cb3948-49b7-46a5-b2df-92239726c45f-kube-api-access-8tlwm\") pod \"limitador-limitador-8588749977-xrhbn\" (UID: \"f8cb3948-49b7-46a5-b2df-92239726c45f\") " pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:00.810821 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:00.810793 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:01.082504 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:01.082386 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xrhbn"] Sep 17 18:11:01.085595 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:11:01.085562 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cb3948_49b7_46a5_b2df_92239726c45f.slice/crio-06102f52fe604927069b90085331706ceb72aad8bc021c769c3631cb3e1a09ca WatchSource:0}: Error finding container 06102f52fe604927069b90085331706ceb72aad8bc021c769c3631cb3e1a09ca: Status 404 returned error can't find the container with id 06102f52fe604927069b90085331706ceb72aad8bc021c769c3631cb3e1a09ca Sep 17 18:11:01.684704 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:01.684665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" event={"ID":"f8cb3948-49b7-46a5-b2df-92239726c45f","Type":"ContainerStarted","Data":"06102f52fe604927069b90085331706ceb72aad8bc021c769c3631cb3e1a09ca"} Sep 17 18:11:02.691035 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:02.690939 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" event={"ID":"f8cb3948-49b7-46a5-b2df-92239726c45f","Type":"ContainerStarted","Data":"9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5"} Sep 17 18:11:02.691035 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:02.691009 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:02.711578 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:02.711533 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" podStartSLOduration=1.389619434 podStartE2EDuration="2.711517104s" totalImagesPullingTime="1.32189767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:02.709489694 +0000 UTC m=+473.773433653" watchObservedRunningTime="2026-09-17 18:11:02.711517104 +0000 UTC m=+473.775461062" Sep 17 18:11:13.697021 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:13.696994 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:16.620230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:16.620196 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xrhbn"] Sep 17 18:11:16.620603 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:16.620395 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" podUID="f8cb3948-49b7-46a5-b2df-92239726c45f" containerName="limitador" containerID="cri-o://9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5" gracePeriod=30 Sep 17 18:11:17.169069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.169046 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:17.207174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.207097 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tlwm\" (UniqueName: \"kubernetes.io/projected/f8cb3948-49b7-46a5-b2df-92239726c45f-kube-api-access-8tlwm\") pod \"f8cb3948-49b7-46a5-b2df-92239726c45f\" (UID: \"f8cb3948-49b7-46a5-b2df-92239726c45f\") " Sep 17 18:11:17.207293 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.207208 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8cb3948-49b7-46a5-b2df-92239726c45f-config-file\") pod \"f8cb3948-49b7-46a5-b2df-92239726c45f\" (UID: \"f8cb3948-49b7-46a5-b2df-92239726c45f\") " Sep 17 18:11:17.207568 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.207546 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cb3948-49b7-46a5-b2df-92239726c45f-config-file" (OuterVolumeSpecName: "config-file") pod "f8cb3948-49b7-46a5-b2df-92239726c45f" (UID: "f8cb3948-49b7-46a5-b2df-92239726c45f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:11:17.209350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.209323 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cb3948-49b7-46a5-b2df-92239726c45f-kube-api-access-8tlwm" (OuterVolumeSpecName: "kube-api-access-8tlwm") pod "f8cb3948-49b7-46a5-b2df-92239726c45f" (UID: "f8cb3948-49b7-46a5-b2df-92239726c45f"). InnerVolumeSpecName "kube-api-access-8tlwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:11:17.308567 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.308540 2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8cb3948-49b7-46a5-b2df-92239726c45f-config-file\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:11:17.308567 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.308566 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tlwm\" (UniqueName: \"kubernetes.io/projected/f8cb3948-49b7-46a5-b2df-92239726c45f-kube-api-access-8tlwm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:11:17.768831 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.768797 2582 generic.go:356] "Generic (PLEG): container finished" podID="f8cb3948-49b7-46a5-b2df-92239726c45f" containerID="9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5" exitCode=0 Sep 17 18:11:17.769234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.768844 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" Sep 17 18:11:17.769234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.768894 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" event={"ID":"f8cb3948-49b7-46a5-b2df-92239726c45f","Type":"ContainerDied","Data":"9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5"} Sep 17 18:11:17.769234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.768949 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xrhbn" event={"ID":"f8cb3948-49b7-46a5-b2df-92239726c45f","Type":"ContainerDied","Data":"06102f52fe604927069b90085331706ceb72aad8bc021c769c3631cb3e1a09ca"} Sep 17 18:11:17.769234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.768972 2582 scope.go:117] "RemoveContainer" containerID="9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5" Sep 17 18:11:17.777444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.777428 2582 scope.go:117] "RemoveContainer" containerID="9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5" Sep 17 18:11:17.777689 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:11:17.777668 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5\": container with ID starting with 9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5 not found: ID does not exist" containerID="9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5" Sep 17 18:11:17.777785 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.777696 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5"} err="failed to get container status \"9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5\": rpc error: code = NotFound desc = could not find container \"9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5\": container with ID starting with 9b152b8b13897edfe178ef2bf1cc5956519c7f0490b66f8b9fb78b80dbe90fa5 not found: ID does not exist" Sep 17 18:11:17.785397 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.785374 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xrhbn"] Sep 17 18:11:17.789480 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:17.789461 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xrhbn"] Sep 17 18:11:19.345315 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:19.345280 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cb3948-49b7-46a5-b2df-92239726c45f" path="/var/lib/kubelet/pods/f8cb3948-49b7-46a5-b2df-92239726c45f/volumes" Sep 17 18:11:22.550134 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.550102 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wxss5"] Sep 17 18:11:22.550525 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.550481 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cb3948-49b7-46a5-b2df-92239726c45f" containerName="limitador" Sep 17 18:11:22.550525 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.550492 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cb3948-49b7-46a5-b2df-92239726c45f" containerName="limitador" Sep 17 18:11:22.550610 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.550565 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cb3948-49b7-46a5-b2df-92239726c45f" containerName="limitador" Sep 17 18:11:22.553288 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.553264 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:22.556605 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.556586 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 17 18:11:22.556907 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.556885 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 18:11:22.556989 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.556951 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nk9hw\"" Sep 17 18:11:22.557608 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.557591 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 18:11:22.565352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.565332 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wxss5"] Sep 17 18:11:22.654410 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.654381 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22fc0eb0-5968-4e5e-808d-b93e602ab9d6-data\") pod \"postgres-69fb4594cf-wxss5\" (UID: \"22fc0eb0-5968-4e5e-808d-b93e602ab9d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:22.654602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.654422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lshjf\" (UniqueName: \"kubernetes.io/projected/22fc0eb0-5968-4e5e-808d-b93e602ab9d6-kube-api-access-lshjf\") pod \"postgres-69fb4594cf-wxss5\" (UID: \"22fc0eb0-5968-4e5e-808d-b93e602ab9d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:22.755370 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.755337 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22fc0eb0-5968-4e5e-808d-b93e602ab9d6-data\") pod \"postgres-69fb4594cf-wxss5\" (UID: \"22fc0eb0-5968-4e5e-808d-b93e602ab9d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:22.755527 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.755391 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lshjf\" (UniqueName: \"kubernetes.io/projected/22fc0eb0-5968-4e5e-808d-b93e602ab9d6-kube-api-access-lshjf\") pod \"postgres-69fb4594cf-wxss5\" (UID: \"22fc0eb0-5968-4e5e-808d-b93e602ab9d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:22.755684 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.755667 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/22fc0eb0-5968-4e5e-808d-b93e602ab9d6-data\") pod \"postgres-69fb4594cf-wxss5\" (UID: \"22fc0eb0-5968-4e5e-808d-b93e602ab9d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:22.769362 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.769329 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lshjf\" (UniqueName: \"kubernetes.io/projected/22fc0eb0-5968-4e5e-808d-b93e602ab9d6-kube-api-access-lshjf\") pod \"postgres-69fb4594cf-wxss5\" (UID: \"22fc0eb0-5968-4e5e-808d-b93e602ab9d6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:22.864687 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:22.864599 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:23.139056 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:23.139032 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wxss5"] Sep 17 18:11:23.141470 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:11:23.141442 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fc0eb0_5968_4e5e_808d_b93e602ab9d6.slice/crio-212c5ba9f7bcbe740d081a7c29c41652d014a6a498b2b7e4a888388f62522a2f WatchSource:0}: Error finding container 212c5ba9f7bcbe740d081a7c29c41652d014a6a498b2b7e4a888388f62522a2f: Status 404 returned error can't find the container with id 212c5ba9f7bcbe740d081a7c29c41652d014a6a498b2b7e4a888388f62522a2f Sep 17 18:11:23.801012 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:23.800974 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" event={"ID":"22fc0eb0-5968-4e5e-808d-b93e602ab9d6","Type":"ContainerStarted","Data":"212c5ba9f7bcbe740d081a7c29c41652d014a6a498b2b7e4a888388f62522a2f"} Sep 17 18:11:28.833344 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:28.833301 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" event={"ID":"22fc0eb0-5968-4e5e-808d-b93e602ab9d6","Type":"ContainerStarted","Data":"d965edd27a8b6b9e067f6f703312f3f992ede90ba8100c34845dc72346ebdfe2"} Sep 17 18:11:28.833731 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:28.833362 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:11:28.852259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:28.852203 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" podStartSLOduration=2.046007091 podStartE2EDuration="6.85218736s" totalImagesPullingTime="4.806180269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:28.850807213 +0000 UTC m=+499.914751173" watchObservedRunningTime="2026-09-17 18:11:28.85218736 +0000 UTC m=+499.916131318" Sep 17 18:11:34.877182 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:11:34.877156 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wxss5" Sep 17 18:12:02.735722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.735686 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-vhqfn"] Sep 17 18:12:02.753274 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.753248 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:02.756077 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.756051 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 18:12:02.756369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.756350 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 18:12:02.756741 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.756717 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-vhqfn"] Sep 17 18:12:02.757064 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.757044 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 18:12:02.757169 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.757049 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6gb7f\"" Sep 17 18:12:02.805382 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.805342 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:02.805575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.805422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:02.805575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.805453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjjg\" (UniqueName: \"kubernetes.io/projected/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-kube-api-access-5pjjg\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:02.906491 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.906445 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:02.906491 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.906493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjjg\" (UniqueName: \"kubernetes.io/projected/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-kube-api-access-5pjjg\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:02.906766 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:12:02.906583 2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 17 18:12:02.906766 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.906606 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:02.906766 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:12:02.906645 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert podName:9ee9f6a0-73cd-49f7-92c2-04362ffc04a3 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:03.406626027 +0000 UTC m=+534.470569963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert") pod "maas-controller-5894d6dbc9-vhqfn" (UID: "9ee9f6a0-73cd-49f7-92c2-04362ffc04a3") : secret "maas-controller-webhook-cert" not found Sep 17 18:12:02.906766 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:12:02.906696 2582 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 17 18:12:02.906766 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:12:02.906754 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs podName:9ee9f6a0-73cd-49f7-92c2-04362ffc04a3 nodeName:}" failed. No retries permitted until 2026-09-17 18:12:03.406735886 +0000 UTC m=+534.470679836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs") pod "maas-controller-5894d6dbc9-vhqfn" (UID: "9ee9f6a0-73cd-49f7-92c2-04362ffc04a3") : secret "maas-controller-metrics-tls" not found Sep 17 18:12:02.929440 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.929409 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjjg\" (UniqueName: \"kubernetes.io/projected/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-kube-api-access-5pjjg\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:02.985365 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.985331 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c8bff4649-b86fq"] Sep 17 18:12:02.988105 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:02.988059 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.007099 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.007073 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hvxp\" (UniqueName: \"kubernetes.io/projected/fbb88877-0e7e-443d-afb5-89d9047e0bbf-kube-api-access-2hvxp\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.007222 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.007148 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-webhook-cert\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.007298 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.007257 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-metrics-certs\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.017448 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.017394 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-b86fq"] Sep 17 18:12:03.108095 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.108064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvxp\" (UniqueName: \"kubernetes.io/projected/fbb88877-0e7e-443d-afb5-89d9047e0bbf-kube-api-access-2hvxp\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.108279 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.108162 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-webhook-cert\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.108279 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.108215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-metrics-certs\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.110657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.110633 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-webhook-cert\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.110787 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.110672 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-metrics-certs\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.117875 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.117839 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvxp\" (UniqueName: \"kubernetes.io/projected/fbb88877-0e7e-443d-afb5-89d9047e0bbf-kube-api-access-2hvxp\") pod \"maas-controller-7c8bff4649-b86fq\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.298216 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.298118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:03.410621 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.410587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:03.410772 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.410644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:03.413700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.413677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:03.413700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.413695 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs\") pod \"maas-controller-5894d6dbc9-vhqfn\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:03.574595 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.574568 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-b86fq"] Sep 17 18:12:03.576479 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:12:03.576454 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb88877_0e7e_443d_afb5_89d9047e0bbf.slice/crio-76a459bad06a16031f787313dbcaade9bcd6a3662280de1b6e976608d5b54962 WatchSource:0}: Error finding container 76a459bad06a16031f787313dbcaade9bcd6a3662280de1b6e976608d5b54962: Status 404 returned error can't find the container with id 76a459bad06a16031f787313dbcaade9bcd6a3662280de1b6e976608d5b54962 Sep 17 18:12:03.668888 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.668831 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:03.941774 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:03.941747 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-vhqfn"] Sep 17 18:12:03.942625 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:12:03.942597 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee9f6a0_73cd_49f7_92c2_04362ffc04a3.slice/crio-de9b15ebdb2d950207974732fe14828dee3a33979fd374a88502192622a609e5 WatchSource:0}: Error finding container de9b15ebdb2d950207974732fe14828dee3a33979fd374a88502192622a609e5: Status 404 returned error can't find the container with id de9b15ebdb2d950207974732fe14828dee3a33979fd374a88502192622a609e5 Sep 17 18:12:04.030566 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:04.030530 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-b86fq" event={"ID":"fbb88877-0e7e-443d-afb5-89d9047e0bbf","Type":"ContainerStarted","Data":"76a459bad06a16031f787313dbcaade9bcd6a3662280de1b6e976608d5b54962"} Sep 17 18:12:04.031649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:04.031626 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" event={"ID":"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3","Type":"ContainerStarted","Data":"de9b15ebdb2d950207974732fe14828dee3a33979fd374a88502192622a609e5"} Sep 17 18:12:07.050837 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:07.050801 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-b86fq" event={"ID":"fbb88877-0e7e-443d-afb5-89d9047e0bbf","Type":"ContainerStarted","Data":"e75ee19deeaa017f104832ed552976a2c6d343750bc4849e740c995a50731ecb"} Sep 17 18:12:07.051357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:07.050917 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:07.052499 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:07.052472 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" event={"ID":"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3","Type":"ContainerStarted","Data":"cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad"} Sep 17 18:12:07.052615 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:07.052577 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:07.072793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:07.072750 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c8bff4649-b86fq" podStartSLOduration=2.407306735 podStartE2EDuration="5.072736843s" totalImagesPullingTime="2.665430108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:07.070419204 +0000 UTC m=+538.134363176" watchObservedRunningTime="2026-09-17 18:12:07.072736843 +0000 UTC m=+538.136680801" Sep 17 18:12:07.091484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:07.091438 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" podStartSLOduration=2.780603939 podStartE2EDuration="5.091424296s" totalImagesPullingTime="2.310820357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:07.089395385 +0000 UTC m=+538.153339347" watchObservedRunningTime="2026-09-17 18:12:07.091424296 +0000 UTC m=+538.155368254" Sep 17 18:12:18.059779 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.059747 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:18.060192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.059817 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:12:18.144757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.144718 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-vhqfn"] Sep 17 18:12:18.145028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.144965 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" podUID="9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" containerName="manager" containerID="cri-o://cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad" gracePeriod=30 Sep 17 18:12:18.397625 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.397594 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:18.447943 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.447901 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert\") pod \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " Sep 17 18:12:18.448095 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.447980 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs\") pod \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " Sep 17 18:12:18.448095 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.448011 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pjjg\" (UniqueName: \"kubernetes.io/projected/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-kube-api-access-5pjjg\") pod \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\" (UID: \"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3\") " Sep 17 18:12:18.450283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.450212 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" (UID: "9ee9f6a0-73cd-49f7-92c2-04362ffc04a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:12:18.450435 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.450301 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" (UID: "9ee9f6a0-73cd-49f7-92c2-04362ffc04a3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:12:18.450655 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.450626 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-kube-api-access-5pjjg" (OuterVolumeSpecName: "kube-api-access-5pjjg") pod "9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" (UID: "9ee9f6a0-73cd-49f7-92c2-04362ffc04a3"). InnerVolumeSpecName "kube-api-access-5pjjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:12:18.549282 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.549253 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-webhook-cert\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:12:18.549282 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.549278 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:12:18.549282 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:18.549287 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pjjg\" (UniqueName: \"kubernetes.io/projected/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3-kube-api-access-5pjjg\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:12:19.117198 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.117163 2582 generic.go:356] "Generic (PLEG): container finished" podID="9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" containerID="cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad" exitCode=0 Sep 17 18:12:19.117596 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.117208 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" Sep 17 18:12:19.117596 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.117251 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" event={"ID":"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3","Type":"ContainerDied","Data":"cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad"} Sep 17 18:12:19.117596 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.117333 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5894d6dbc9-vhqfn" event={"ID":"9ee9f6a0-73cd-49f7-92c2-04362ffc04a3","Type":"ContainerDied","Data":"de9b15ebdb2d950207974732fe14828dee3a33979fd374a88502192622a609e5"} Sep 17 18:12:19.117596 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.117356 2582 scope.go:117] "RemoveContainer" containerID="cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad" Sep 17 18:12:19.126463 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.126445 2582 scope.go:117] "RemoveContainer" containerID="cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad" Sep 17 18:12:19.126720 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:12:19.126701 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad\": container with ID starting with cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad not found: ID does not exist" containerID="cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad" Sep 17 18:12:19.126789 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.126733 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad"} err="failed to get container status \"cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad\": rpc error: code = NotFound desc = could not find container \"cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad\": container with ID starting with cf41bd86ed2a60df41b265c5196298ccb8850d93ddf2cf6c190157afd790dbad not found: ID does not exist" Sep 17 18:12:19.139321 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.139290 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-vhqfn"] Sep 17 18:12:19.143720 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.143698 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5894d6dbc9-vhqfn"] Sep 17 18:12:19.345023 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:19.344986 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" path="/var/lib/kubelet/pods/9ee9f6a0-73cd-49f7-92c2-04362ffc04a3/volumes" Sep 17 18:12:40.727946 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.727915 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx"] Sep 17 18:12:40.728422 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.728299 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" containerName="manager" Sep 17 18:12:40.728422 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.728310 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" containerName="manager" Sep 17 18:12:40.728422 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.728385 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee9f6a0-73cd-49f7-92c2-04362ffc04a3" containerName="manager" Sep 17 18:12:40.732983 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.732965 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.736820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.736799 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sqjt5\"" Sep 17 18:12:40.736965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.736888 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 18:12:40.736965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.736924 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 18:12:40.743882 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.743586 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx"] Sep 17 18:12:40.851449 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.851410 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.851632 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.851586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fxp\" (UniqueName: \"kubernetes.io/projected/9dca8211-671d-4466-a367-dcfc192a9718-kube-api-access-j9fxp\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.851632 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.851625 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-metrics-certs\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.851793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.851771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dca8211-671d-4466-a367-dcfc192a9718-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.953299 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.953267 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.953299 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.953301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fxp\" (UniqueName: \"kubernetes.io/projected/9dca8211-671d-4466-a367-dcfc192a9718-kube-api-access-j9fxp\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.953537 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.953322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-metrics-certs\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.953537 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.953371 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dca8211-671d-4466-a367-dcfc192a9718-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.954217 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.954187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dca8211-671d-4466-a367-dcfc192a9718-openshift-service-ca\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.955949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.955924 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-metrics-certs\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.956067 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.955988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-maas-api-tls\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:40.961995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:40.961976 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fxp\" (UniqueName: \"kubernetes.io/projected/9dca8211-671d-4466-a367-dcfc192a9718-kube-api-access-j9fxp\") pod \"maas-api-5ffb5cf8dc-t4grx\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:41.052736 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.052662 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:41.149276 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.149248 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk"] Sep 17 18:12:41.153488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.153466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:41.157965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.157939 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 17 18:12:41.171598 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.171566 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk"] Sep 17 18:12:41.174240 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.174212 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8k6rh"] Sep 17 18:12:41.177491 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.177470 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:41.180463 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.180434 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pzqrq\"" Sep 17 18:12:41.194402 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.194377 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8k6rh"] Sep 17 18:12:41.255795 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.255765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7g88\" (UniqueName: \"kubernetes.io/projected/edfdc12f-f3c4-426e-86fc-8cb583abc601-kube-api-access-c7g88\") pod \"payload-pre-processing-59fbc47bf8-kkjdk\" (UID: \"edfdc12f-f3c4-426e-86fc-8cb583abc601\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:41.255973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.255913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edfdc12f-f3c4-426e-86fc-8cb583abc601-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kkjdk\" (UID: \"edfdc12f-f3c4-426e-86fc-8cb583abc601\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:41.330201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.330169 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx"] Sep 17 18:12:41.332048 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:12:41.332011 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dca8211_671d_4466_a367_dcfc192a9718.slice/crio-ee6db1d08e4959e2da6a191f2bb676832d904d7721d95c5c27ec77a0f55b398c WatchSource:0}: Error finding container ee6db1d08e4959e2da6a191f2bb676832d904d7721d95c5c27ec77a0f55b398c: Status 404 returned error can't find the container with id ee6db1d08e4959e2da6a191f2bb676832d904d7721d95c5c27ec77a0f55b398c Sep 17 18:12:41.357258 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.357230 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6927897e-3956-4585-8de2-bea9c1384dee-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8k6rh\" (UID: \"6927897e-3956-4585-8de2-bea9c1384dee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:41.357404 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.357271 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edfdc12f-f3c4-426e-86fc-8cb583abc601-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kkjdk\" (UID: \"edfdc12f-f3c4-426e-86fc-8cb583abc601\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:41.357404 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.357293 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bprlf\" (UniqueName: \"kubernetes.io/projected/6927897e-3956-4585-8de2-bea9c1384dee-kube-api-access-bprlf\") pod \"payload-processing-5b54b7fbb7-8k6rh\" (UID: \"6927897e-3956-4585-8de2-bea9c1384dee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:41.357404 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.357389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7g88\" (UniqueName: \"kubernetes.io/projected/edfdc12f-f3c4-426e-86fc-8cb583abc601-kube-api-access-c7g88\") pod \"payload-pre-processing-59fbc47bf8-kkjdk\" (UID: \"edfdc12f-f3c4-426e-86fc-8cb583abc601\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:41.358020 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.357998 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edfdc12f-f3c4-426e-86fc-8cb583abc601-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kkjdk\" (UID: \"edfdc12f-f3c4-426e-86fc-8cb583abc601\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:41.367076 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.367056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7g88\" (UniqueName: \"kubernetes.io/projected/edfdc12f-f3c4-426e-86fc-8cb583abc601-kube-api-access-c7g88\") pod \"payload-pre-processing-59fbc47bf8-kkjdk\" (UID: \"edfdc12f-f3c4-426e-86fc-8cb583abc601\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:41.458943 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.458910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6927897e-3956-4585-8de2-bea9c1384dee-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8k6rh\" (UID: \"6927897e-3956-4585-8de2-bea9c1384dee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:41.459140 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.458997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bprlf\" (UniqueName: \"kubernetes.io/projected/6927897e-3956-4585-8de2-bea9c1384dee-kube-api-access-bprlf\") pod \"payload-processing-5b54b7fbb7-8k6rh\" (UID: \"6927897e-3956-4585-8de2-bea9c1384dee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:41.459701 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.459682 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6927897e-3956-4585-8de2-bea9c1384dee-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8k6rh\" (UID: \"6927897e-3956-4585-8de2-bea9c1384dee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:41.465791 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.465759 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:41.480671 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.480648 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bprlf\" (UniqueName: \"kubernetes.io/projected/6927897e-3956-4585-8de2-bea9c1384dee-kube-api-access-bprlf\") pod \"payload-processing-5b54b7fbb7-8k6rh\" (UID: \"6927897e-3956-4585-8de2-bea9c1384dee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:41.487531 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.487509 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:41.743796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.743763 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk"] Sep 17 18:12:41.746906 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:12:41.746853 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfdc12f_f3c4_426e_86fc_8cb583abc601.slice/crio-11c13ddf3598c9e8a89dd132030024470d2826fc02a714de35b1215b8b14512a WatchSource:0}: Error finding container 11c13ddf3598c9e8a89dd132030024470d2826fc02a714de35b1215b8b14512a: Status 404 returned error can't find the container with id 11c13ddf3598c9e8a89dd132030024470d2826fc02a714de35b1215b8b14512a Sep 17 18:12:41.779321 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:41.779294 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8k6rh"] Sep 17 18:12:41.781127 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:12:41.781097 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6927897e_3956_4585_8de2_bea9c1384dee.slice/crio-038b2024d5990923af5f2235a53ac0475c4bf12801b28f2d51a33a0d76e613e9 WatchSource:0}: Error finding container 038b2024d5990923af5f2235a53ac0475c4bf12801b28f2d51a33a0d76e613e9: Status 404 returned error can't find the container with id 038b2024d5990923af5f2235a53ac0475c4bf12801b28f2d51a33a0d76e613e9 Sep 17 18:12:42.236842 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:42.236793 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" event={"ID":"6927897e-3956-4585-8de2-bea9c1384dee","Type":"ContainerStarted","Data":"038b2024d5990923af5f2235a53ac0475c4bf12801b28f2d51a33a0d76e613e9"} Sep 17 18:12:42.244894 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:42.239813 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" event={"ID":"9dca8211-671d-4466-a367-dcfc192a9718","Type":"ContainerStarted","Data":"ee6db1d08e4959e2da6a191f2bb676832d904d7721d95c5c27ec77a0f55b398c"} Sep 17 18:12:42.244894 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:42.241330 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" event={"ID":"edfdc12f-f3c4-426e-86fc-8cb583abc601","Type":"ContainerStarted","Data":"11c13ddf3598c9e8a89dd132030024470d2826fc02a714de35b1215b8b14512a"} Sep 17 18:12:44.272912 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.272798 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" event={"ID":"6927897e-3956-4585-8de2-bea9c1384dee","Type":"ContainerStarted","Data":"bdfea9bd995ccb2f9817caa68cf1dfba068a7c02f1457d2c7fd5f77f2fdc7ffa"} Sep 17 18:12:44.273356 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.272928 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:44.274778 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.274755 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" event={"ID":"9dca8211-671d-4466-a367-dcfc192a9718","Type":"ContainerStarted","Data":"8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572"} Sep 17 18:12:44.274947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.274814 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:44.276485 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.276461 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" event={"ID":"edfdc12f-f3c4-426e-86fc-8cb583abc601","Type":"ContainerStarted","Data":"875282ab8abf4cd4d75452ef04c3a8249e27fd71c02bee431cf0f198def2f453"} Sep 17 18:12:44.276571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.276536 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:44.316382 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.316333 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" podStartSLOduration=1.075333075 podStartE2EDuration="3.316314352s" totalImagesPullingTime="2.240981277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:44.29417505 +0000 UTC m=+575.358119010" watchObservedRunningTime="2026-09-17 18:12:44.316314352 +0000 UTC m=+575.380258308" Sep 17 18:12:44.316955 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.316917 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" podStartSLOduration=1.044217046 podStartE2EDuration="3.316910198s" totalImagesPullingTime="2.272693152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:44.315228818 +0000 UTC m=+575.379172775" watchObservedRunningTime="2026-09-17 18:12:44.316910198 +0000 UTC m=+575.380854159" Sep 17 18:12:44.343499 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:44.343441 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" podStartSLOduration=1.6556209370000001 podStartE2EDuration="4.343420763s" totalImagesPullingTime="2.687799826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:44.342282984 +0000 UTC m=+575.406226943" watchObservedRunningTime="2026-09-17 18:12:44.343420763 +0000 UTC m=+575.407364762" Sep 17 18:12:50.289948 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:50.289923 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:12:50.627452 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:50.627367 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6v2jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:12:50.627452 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:50.627421 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" podUID="0f850692-e665-4012-bc51-ab3bd9c42ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 17 18:12:51.316427 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:51.316395 2582 generic.go:356] "Generic (PLEG): container finished" podID="0f850692-e665-4012-bc51-ab3bd9c42ab0" containerID="918a325152081573d6017b4a7818cbd6c0ae66fccc291143f8d422f0a8043656" exitCode=0 Sep 17 18:12:51.316925 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:51.316457 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" event={"ID":"0f850692-e665-4012-bc51-ab3bd9c42ab0","Type":"ContainerDied","Data":"918a325152081573d6017b4a7818cbd6c0ae66fccc291143f8d422f0a8043656"} Sep 17 18:12:51.316925 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:51.316844 2582 scope.go:117] "RemoveContainer" containerID="918a325152081573d6017b4a7818cbd6c0ae66fccc291143f8d422f0a8043656" Sep 17 18:12:51.317927 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:51.317858 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 18:12:51.318139 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:51.318116 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 18:12:51.318285 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:51.318266 2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 17 18:12:52.324058 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:52.324021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" event={"ID":"0f850692-e665-4012-bc51-ab3bd9c42ab0","Type":"ContainerStarted","Data":"71434b62e384c227797182ed20f5030ec9b675eb5fe2256b291aa7061bac123d"} Sep 17 18:12:53.271262 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:53.271227 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:12:53.271458 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:53.271409 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6v2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:12:53.271532 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:53.271469 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" podUID="0f850692-e665-4012-bc51-ab3bd9c42ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 17 18:12:54.271165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:54.271133 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6v2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:12:54.271561 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:54.271189 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" podUID="0f850692-e665-4012-bc51-ab3bd9c42ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 17 18:12:55.271932 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:55.271890 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6v2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:12:55.272317 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:55.271956 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" podUID="0f850692-e665-4012-bc51-ab3bd9c42ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 17 18:12:55.285261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:55.285233 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kkjdk" Sep 17 18:12:55.285417 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:55.285405 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-8k6rh" Sep 17 18:12:56.271294 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:56.271254 2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-6v2jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Sep 17 18:12:56.271509 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:56.271361 2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" podUID="0f850692-e665-4012-bc51-ab3bd9c42ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Sep 17 18:12:57.297428 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:57.297393 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:12:57.297829 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:57.297804 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:12:57.298443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:12:57.298423 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-6v2jt" Sep 17 18:13:11.245677 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:11.245652 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:13:11.249077 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:11.249053 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:13:11.268966 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:11.268943 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:13:11.272117 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:11.272098 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:13:32.264775 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.264681 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs"] Sep 17 18:13:32.268493 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.268469 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.275243 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.275064 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 18:13:32.276840 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.276393 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 18:13:32.278573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.278542 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 17 18:13:32.278688 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.278615 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k9rwt\"" Sep 17 18:13:32.296531 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.296502 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs"] Sep 17 18:13:32.332354 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.332325 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.332492 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.332360 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm8pr\" (UniqueName: \"kubernetes.io/projected/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-kube-api-access-rm8pr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.332492 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.332439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.332571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.332501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.332571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.332539 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.332653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.332637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433094 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433065 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433249 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433115 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433249 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm8pr\" (UniqueName: \"kubernetes.io/projected/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-kube-api-access-rm8pr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433284 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433450 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433353 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433542 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433506 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433762 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433742 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.433902 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.433795 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.435537 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.435515 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.435708 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.435691 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.442758 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.442737 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm8pr\" (UniqueName: \"kubernetes.io/projected/0bbaa877-0b3a-43a2-89a5-bafbf307e4f0-kube-api-access-rm8pr\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs\" (UID: \"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.580201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.580123 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:32.859921 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.859896 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs"] Sep 17 18:13:32.862294 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:13:32.862245 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbaa877_0b3a_43a2_89a5_bafbf307e4f0.slice/crio-955ce7c4536c6ef3c5fa1bed7e4e7a508ab116012ee186bf729664d64c53870a WatchSource:0}: Error finding container 955ce7c4536c6ef3c5fa1bed7e4e7a508ab116012ee186bf729664d64c53870a: Status 404 returned error can't find the container with id 955ce7c4536c6ef3c5fa1bed7e4e7a508ab116012ee186bf729664d64c53870a Sep 17 18:13:32.863624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:32.863605 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:13:33.545825 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:33.545770 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" event={"ID":"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0","Type":"ContainerStarted","Data":"955ce7c4536c6ef3c5fa1bed7e4e7a508ab116012ee186bf729664d64c53870a"} Sep 17 18:13:34.552189 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:34.552152 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" event={"ID":"0bbaa877-0b3a-43a2-89a5-bafbf307e4f0","Type":"ContainerStarted","Data":"91cc4afeb3d24e4dff977716b392bc39b8fdd1d99415a94f6f885993502c618f"} Sep 17 18:13:34.571484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:34.571435 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" podStartSLOduration=1.579708493 podStartE2EDuration="2.571422332s" totalImagesPullingTime="991.713839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:34.569664276 +0000 UTC m=+625.633608236" watchObservedRunningTime="2026-09-17 18:13:34.571422332 +0000 UTC m=+625.635366289" Sep 17 18:13:42.581103 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:42.581069 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:42.581502 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:42.581137 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:42.593724 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:42.593699 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:13:42.613358 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:13:42.613332 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs" Sep 17 18:16:21.236504 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.236469 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7640-e893e6"] Sep 17 18:16:21.240074 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.240052 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7640-e893e6" Sep 17 18:16:21.242728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.242639 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5zgf\"" Sep 17 18:16:21.246902 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.246854 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7640-e893e6"] Sep 17 18:16:21.299311 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.299274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqvt\" (UniqueName: \"kubernetes.io/projected/abaedb5a-7b12-4d68-b7b6-25011073ff44-kube-api-access-psqvt\") pod \"test-curl-7640-e893e6\" (UID: \"abaedb5a-7b12-4d68-b7b6-25011073ff44\") " pod="opendatahub/test-curl-7640-e893e6" Sep 17 18:16:21.400367 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.400327 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqvt\" (UniqueName: \"kubernetes.io/projected/abaedb5a-7b12-4d68-b7b6-25011073ff44-kube-api-access-psqvt\") pod \"test-curl-7640-e893e6\" (UID: \"abaedb5a-7b12-4d68-b7b6-25011073ff44\") " pod="opendatahub/test-curl-7640-e893e6" Sep 17 18:16:21.412517 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.412492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqvt\" (UniqueName: \"kubernetes.io/projected/abaedb5a-7b12-4d68-b7b6-25011073ff44-kube-api-access-psqvt\") pod \"test-curl-7640-e893e6\" (UID: \"abaedb5a-7b12-4d68-b7b6-25011073ff44\") " pod="opendatahub/test-curl-7640-e893e6" Sep 17 18:16:21.551797 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.551705 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7640-e893e6" Sep 17 18:16:21.824478 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:21.824451 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7640-e893e6"] Sep 17 18:16:21.826827 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:21.826799 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabaedb5a_7b12_4d68_b7b6_25011073ff44.slice/crio-1aef7f2aa57dd65a40acb3a08110c163bff3920a0b05f4f20ceecd32167f5804 WatchSource:0}: Error finding container 1aef7f2aa57dd65a40acb3a08110c163bff3920a0b05f4f20ceecd32167f5804: Status 404 returned error can't find the container with id 1aef7f2aa57dd65a40acb3a08110c163bff3920a0b05f4f20ceecd32167f5804 Sep 17 18:16:22.465990 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:22.465908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7640-e893e6" event={"ID":"abaedb5a-7b12-4d68-b7b6-25011073ff44","Type":"ContainerStarted","Data":"1aef7f2aa57dd65a40acb3a08110c163bff3920a0b05f4f20ceecd32167f5804"} Sep 17 18:16:24.015046 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.015012 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5"] Sep 17 18:16:24.026508 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.026461 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.029360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.029020 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c31adf-serving-cert\"" Sep 17 18:16:24.029360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.029186 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5"] Sep 17 18:16:24.131417 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.131377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633d73b-d1c9-431a-ae39-b3aa4b81a431-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.131417 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.131424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-maas-api-tls\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.131676 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.131455 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqhs9\" (UniqueName: \"kubernetes.io/projected/8633d73b-d1c9-431a-ae39-b3aa4b81a431-kube-api-access-mqhs9\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.131676 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.131581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-metrics-certs\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.232577 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.232541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-metrics-certs\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.232929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.232627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633d73b-d1c9-431a-ae39-b3aa4b81a431-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.232929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.232649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-maas-api-tls\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.232929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.232672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqhs9\" (UniqueName: \"kubernetes.io/projected/8633d73b-d1c9-431a-ae39-b3aa4b81a431-kube-api-access-mqhs9\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.233478 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.233458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633d73b-d1c9-431a-ae39-b3aa4b81a431-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.234970 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.234947 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-metrics-certs\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.235118 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.235098 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-maas-api-tls\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.244037 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.244005 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqhs9\" (UniqueName: \"kubernetes.io/projected/8633d73b-d1c9-431a-ae39-b3aa4b81a431-kube-api-access-mqhs9\") pod \"maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.346294 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.346265 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:24.484712 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.484670 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7640-e893e6" event={"ID":"abaedb5a-7b12-4d68-b7b6-25011073ff44","Type":"ContainerStarted","Data":"773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7"} Sep 17 18:16:24.502896 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.502797 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7640-e893e6" podStartSLOduration=1.052306806 podStartE2EDuration="3.50277152s" totalImagesPullingTime="2.450464714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:24.501170804 +0000 UTC m=+795.565114763" watchObservedRunningTime="2026-09-17 18:16:24.50277152 +0000 UTC m=+795.566715479" Sep 17 18:16:24.637217 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.633305 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5"] Sep 17 18:16:24.645579 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:24.645524 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8633d73b_d1c9_431a_ae39_b3aa4b81a431.slice/crio-95b5072e32b42388400ecacfae8523c10c43ec60fce63b7f03fd09c9cc997941 WatchSource:0}: Error finding container 95b5072e32b42388400ecacfae8523c10c43ec60fce63b7f03fd09c9cc997941: Status 404 returned error can't find the container with id 95b5072e32b42388400ecacfae8523c10c43ec60fce63b7f03fd09c9cc997941 Sep 17 18:16:24.795816 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.795774 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4"] Sep 17 18:16:24.801373 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.801348 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:24.803820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.803793 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c31adf\"" Sep 17 18:16:24.808794 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.808770 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4"] Sep 17 18:16:24.813245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.813151 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml"] Sep 17 18:16:24.817372 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.817351 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:24.820072 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.820048 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c31adf-dockercfg-xkxxb\"" Sep 17 18:16:24.825203 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.825039 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml"] Sep 17 18:16:24.849671 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.849621 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpbr\" (UniqueName: \"kubernetes.io/projected/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-kube-api-access-xzpbr\") pod \"payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4\" (UID: \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:24.849894 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.849681 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4\" (UID: \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:24.950588 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.950548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017fb4d2-88d2-4585-a390-3cd31221516a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml\" (UID: \"017fb4d2-88d2-4585-a390-3cd31221516a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:24.950818 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.950609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q92pw\" (UniqueName: \"kubernetes.io/projected/017fb4d2-88d2-4585-a390-3cd31221516a-kube-api-access-q92pw\") pod \"payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml\" (UID: \"017fb4d2-88d2-4585-a390-3cd31221516a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:24.950818 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.950730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpbr\" (UniqueName: \"kubernetes.io/projected/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-kube-api-access-xzpbr\") pod \"payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4\" (UID: \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:24.950818 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.950765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4\" (UID: \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:24.951753 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.951727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4\" (UID: \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:24.961044 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:24.961015 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpbr\" (UniqueName: \"kubernetes.io/projected/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-kube-api-access-xzpbr\") pod \"payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4\" (UID: \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:25.052498 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.052441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017fb4d2-88d2-4585-a390-3cd31221516a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml\" (UID: \"017fb4d2-88d2-4585-a390-3cd31221516a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:25.052498 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.052502 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q92pw\" (UniqueName: \"kubernetes.io/projected/017fb4d2-88d2-4585-a390-3cd31221516a-kube-api-access-q92pw\") pod \"payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml\" (UID: \"017fb4d2-88d2-4585-a390-3cd31221516a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:25.053474 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.053446 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017fb4d2-88d2-4585-a390-3cd31221516a-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml\" (UID: \"017fb4d2-88d2-4585-a390-3cd31221516a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:25.062032 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.061995 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q92pw\" (UniqueName: \"kubernetes.io/projected/017fb4d2-88d2-4585-a390-3cd31221516a-kube-api-access-q92pw\") pod \"payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml\" (UID: \"017fb4d2-88d2-4585-a390-3cd31221516a\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:25.108342 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.108305 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7640-e893e6"] Sep 17 18:16:25.111174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.111139 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7640-e893e6"] Sep 17 18:16:25.116767 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.116744 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:25.132118 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.131750 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:25.310973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.310936 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4"] Sep 17 18:16:25.311654 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.311633 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abaedb5a-7b12-4d68-b7b6-25011073ff44" containerName="test-curl-7640-e893e6" Sep 17 18:16:25.311704 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.311659 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="abaedb5a-7b12-4d68-b7b6-25011073ff44" containerName="test-curl-7640-e893e6" Sep 17 18:16:25.311792 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.311779 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="abaedb5a-7b12-4d68-b7b6-25011073ff44" containerName="test-curl-7640-e893e6" Sep 17 18:16:25.315236 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.315209 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.317592 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.317563 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b18433-serving-cert\"" Sep 17 18:16:25.325732 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.325656 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4"] Sep 17 18:16:25.359295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.359263 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56258a1-d9fa-43ee-8495-88e71277f744-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.359461 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.359316 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxts9\" (UniqueName: \"kubernetes.io/projected/e56258a1-d9fa-43ee-8495-88e71277f744-kube-api-access-zxts9\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.359461 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.359414 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.359461 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.359439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-metrics-certs\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.426387 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.426359 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4"] Sep 17 18:16:25.428896 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:25.428854 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9c824f_ab2c_4a7c_ad45_7d2f8ac34c71.slice/crio-be5c9da9c8a1cc28be54df8149ffad4070c91e1b0ad6cf0ba17c3c38a777adc5 WatchSource:0}: Error finding container be5c9da9c8a1cc28be54df8149ffad4070c91e1b0ad6cf0ba17c3c38a777adc5: Status 404 returned error can't find the container with id be5c9da9c8a1cc28be54df8149ffad4070c91e1b0ad6cf0ba17c3c38a777adc5 Sep 17 18:16:25.460954 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.460929 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.461083 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.460984 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-metrics-certs\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.461083 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.461044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56258a1-d9fa-43ee-8495-88e71277f744-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.461319 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.461092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxts9\" (UniqueName: \"kubernetes.io/projected/e56258a1-d9fa-43ee-8495-88e71277f744-kube-api-access-zxts9\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.461745 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.461719 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml"] Sep 17 18:16:25.461853 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.461802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56258a1-d9fa-43ee-8495-88e71277f744-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.463534 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.463512 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-metrics-certs\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.463645 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.463631 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.466114 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:25.466090 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017fb4d2_88d2_4585_a390_3cd31221516a.slice/crio-1fac063c429a33499a0a33b7197694bf32e3d33044779bd839e9f17faf4878ae WatchSource:0}: Error finding container 1fac063c429a33499a0a33b7197694bf32e3d33044779bd839e9f17faf4878ae: Status 404 returned error can't find the container with id 1fac063c429a33499a0a33b7197694bf32e3d33044779bd839e9f17faf4878ae Sep 17 18:16:25.472221 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.472201 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxts9\" (UniqueName: \"kubernetes.io/projected/e56258a1-d9fa-43ee-8495-88e71277f744-kube-api-access-zxts9\") pod \"maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.493446 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.493416 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" event={"ID":"8633d73b-d1c9-431a-ae39-b3aa4b81a431","Type":"ContainerStarted","Data":"a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c"} Sep 17 18:16:25.493569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.493482 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" event={"ID":"8633d73b-d1c9-431a-ae39-b3aa4b81a431","Type":"ContainerStarted","Data":"95b5072e32b42388400ecacfae8523c10c43ec60fce63b7f03fd09c9cc997941"} Sep 17 18:16:25.493569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.493516 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:25.494725 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.494704 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" event={"ID":"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71","Type":"ContainerStarted","Data":"be5c9da9c8a1cc28be54df8149ffad4070c91e1b0ad6cf0ba17c3c38a777adc5"} Sep 17 18:16:25.495914 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.495890 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" event={"ID":"017fb4d2-88d2-4585-a390-3cd31221516a","Type":"ContainerStarted","Data":"1fac063c429a33499a0a33b7197694bf32e3d33044779bd839e9f17faf4878ae"} Sep 17 18:16:25.496023 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:25.496011 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7640-e893e6" container="test-curl-7640-e893e6" Sep 17 18:16:25.514368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.514259 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" podStartSLOduration=2.048543706 podStartE2EDuration="2.514235864s" totalImagesPullingTime="465.692158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:25.513125146 +0000 UTC m=+796.577069105" watchObservedRunningTime="2026-09-17 18:16:25.514235864 +0000 UTC m=+796.578179824" Sep 17 18:16:25.629640 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.629608 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:25.915305 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:25.915271 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4"] Sep 17 18:16:25.917959 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:25.917915 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56258a1_d9fa_43ee_8495_88e71277f744.slice/crio-630898fb038cab5d60d301b3ad7d92a71ba7a810f34b4553d4098b43b747f4c9 WatchSource:0}: Error finding container 630898fb038cab5d60d301b3ad7d92a71ba7a810f34b4553d4098b43b747f4c9: Status 404 returned error can't find the container with id 630898fb038cab5d60d301b3ad7d92a71ba7a810f34b4553d4098b43b747f4c9 Sep 17 18:16:26.103571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.103474 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm"] Sep 17 18:16:26.109145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.109118 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:26.112017 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.111991 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b18433-dockercfg-skskz\"" Sep 17 18:16:26.112147 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.111992 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b18433\"" Sep 17 18:16:26.118065 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.118036 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm"] Sep 17 18:16:26.169916 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.169881 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b42d62b-e740-4a13-8864-784ad5829e6a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm\" (UID: \"9b42d62b-e740-4a13-8864-784ad5829e6a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:26.170103 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.169956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbtfc\" (UniqueName: \"kubernetes.io/projected/9b42d62b-e740-4a13-8864-784ad5829e6a-kube-api-access-bbtfc\") pod \"payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm\" (UID: \"9b42d62b-e740-4a13-8864-784ad5829e6a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:26.270879 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.270762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b42d62b-e740-4a13-8864-784ad5829e6a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm\" (UID: \"9b42d62b-e740-4a13-8864-784ad5829e6a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:26.270879 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.270850 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbtfc\" (UniqueName: \"kubernetes.io/projected/9b42d62b-e740-4a13-8864-784ad5829e6a-kube-api-access-bbtfc\") pod \"payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm\" (UID: \"9b42d62b-e740-4a13-8864-784ad5829e6a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:26.271722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.271700 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b42d62b-e740-4a13-8864-784ad5829e6a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm\" (UID: \"9b42d62b-e740-4a13-8864-784ad5829e6a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:26.280305 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.280250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbtfc\" (UniqueName: \"kubernetes.io/projected/9b42d62b-e740-4a13-8864-784ad5829e6a-kube-api-access-bbtfc\") pod \"payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm\" (UID: \"9b42d62b-e740-4a13-8864-784ad5829e6a\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:26.408191 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.408158 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b"] Sep 17 18:16:26.412784 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.412760 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.416600 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.416419 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-5d8053-serving-cert\"" Sep 17 18:16:26.424194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.422417 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:26.424194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.424073 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b"] Sep 17 18:16:26.474018 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.473450 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce958664-dfe0-454a-aae9-bd23d9844de0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.474018 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.473509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgttd\" (UniqueName: \"kubernetes.io/projected/ce958664-dfe0-454a-aae9-bd23d9844de0-kube-api-access-kgttd\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.474018 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.473590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-metrics-certs\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.474018 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.473638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.511350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.510502 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" event={"ID":"e56258a1-d9fa-43ee-8495-88e71277f744","Type":"ContainerStarted","Data":"7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f"} Sep 17 18:16:26.511350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.510570 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" event={"ID":"e56258a1-d9fa-43ee-8495-88e71277f744","Type":"ContainerStarted","Data":"630898fb038cab5d60d301b3ad7d92a71ba7a810f34b4553d4098b43b747f4c9"} Sep 17 18:16:26.511350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.511065 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:26.513304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.512735 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" event={"ID":"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71","Type":"ContainerStarted","Data":"0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd"} Sep 17 18:16:26.513304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.513264 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:26.517800 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.517716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" event={"ID":"017fb4d2-88d2-4585-a390-3cd31221516a","Type":"ContainerStarted","Data":"79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04"} Sep 17 18:16:26.517800 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.517759 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:26.518002 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.517836 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7640-e893e6" podUID="abaedb5a-7b12-4d68-b7b6-25011073ff44" containerName="test-curl-7640-e893e6" containerID="cri-o://773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7" gracePeriod=2 Sep 17 18:16:26.533572 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.533508 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" podStartSLOduration=1.214373882 podStartE2EDuration="1.533483906s" totalImagesPullingTime="319.110024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:26.53100057 +0000 UTC m=+797.594944554" watchObservedRunningTime="2026-09-17 18:16:26.533483906 +0000 UTC m=+797.597428073" Sep 17 18:16:26.554067 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.554023 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" podStartSLOduration=2.554006833 podStartE2EDuration="2.554006833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:26.551620584 +0000 UTC m=+797.615564542" watchObservedRunningTime="2026-09-17 18:16:26.554006833 +0000 UTC m=+797.617950791" Sep 17 18:16:26.571915 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.571853 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" podStartSLOduration=2.57183809 podStartE2EDuration="2.57183809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:26.570338745 +0000 UTC m=+797.634282703" watchObservedRunningTime="2026-09-17 18:16:26.57183809 +0000 UTC m=+797.635782048" Sep 17 18:16:26.576124 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.575235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgttd\" (UniqueName: \"kubernetes.io/projected/ce958664-dfe0-454a-aae9-bd23d9844de0-kube-api-access-kgttd\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.576124 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.575279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-metrics-certs\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.576124 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.575315 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.576124 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:26.575428 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-serving-cert: secret "maas-api-e2e-worker-w0-5d8053-serving-cert" not found Sep 17 18:16:26.576124 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:26.575497 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls podName:ce958664-dfe0-454a-aae9-bd23d9844de0 nodeName:}" failed. No retries permitted until 2026-09-17 18:16:27.075477285 +0000 UTC m=+798.139421222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls") pod "maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" (UID: "ce958664-dfe0-454a-aae9-bd23d9844de0") : secret "maas-api-e2e-worker-w0-5d8053-serving-cert" not found Sep 17 18:16:26.576124 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.575661 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce958664-dfe0-454a-aae9-bd23d9844de0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.576436 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.576353 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce958664-dfe0-454a-aae9-bd23d9844de0-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.578224 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.578203 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-metrics-certs\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.585581 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.585559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgttd\" (UniqueName: \"kubernetes.io/projected/ce958664-dfe0-454a-aae9-bd23d9844de0-kube-api-access-kgttd\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:26.722405 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:26.722378 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm"] Sep 17 18:16:26.724853 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:26.724827 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b42d62b_e740_4a13_8864_784ad5829e6a.slice/crio-82531838fe091f986df4571df6f657654a3e6d3aaf4b8c13555382009b9c11df WatchSource:0}: Error finding container 82531838fe091f986df4571df6f657654a3e6d3aaf4b8c13555382009b9c11df: Status 404 returned error can't find the container with id 82531838fe091f986df4571df6f657654a3e6d3aaf4b8c13555382009b9c11df Sep 17 18:16:27.089778 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:27.089703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:27.091999 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:27.091970 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls\") pod \"maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:27.334357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:27.334325 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:27.526578 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:27.526474 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" event={"ID":"9b42d62b-e740-4a13-8864-784ad5829e6a","Type":"ContainerStarted","Data":"e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779"} Sep 17 18:16:27.526578 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:27.526537 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" event={"ID":"9b42d62b-e740-4a13-8864-784ad5829e6a","Type":"ContainerStarted","Data":"82531838fe091f986df4571df6f657654a3e6d3aaf4b8c13555382009b9c11df"} Sep 17 18:16:27.526832 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:27.526797 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:27.547396 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:27.547339 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" podStartSLOduration=1.547321464 podStartE2EDuration="1.547321464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:27.545379745 +0000 UTC m=+798.609323724" watchObservedRunningTime="2026-09-17 18:16:27.547321464 +0000 UTC m=+798.611265423" Sep 17 18:16:27.645805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:27.645777 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b"] Sep 17 18:16:27.648222 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:27.648179 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce958664_dfe0_454a_aae9_bd23d9844de0.slice/crio-3d9d5b3b08349ffe8bc6e761bfc233a72ac8ce13824d18e34708e5b8160c7240 WatchSource:0}: Error finding container 3d9d5b3b08349ffe8bc6e761bfc233a72ac8ce13824d18e34708e5b8160c7240: Status 404 returned error can't find the container with id 3d9d5b3b08349ffe8bc6e761bfc233a72ac8ce13824d18e34708e5b8160c7240 Sep 17 18:16:28.536551 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:28.536456 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" event={"ID":"ce958664-dfe0-454a-aae9-bd23d9844de0","Type":"ContainerStarted","Data":"9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9"} Sep 17 18:16:28.536551 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:28.536511 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" event={"ID":"ce958664-dfe0-454a-aae9-bd23d9844de0","Type":"ContainerStarted","Data":"3d9d5b3b08349ffe8bc6e761bfc233a72ac8ce13824d18e34708e5b8160c7240"} Sep 17 18:16:28.537063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:28.536907 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:28.555940 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:28.555208 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" podStartSLOduration=2.1501899 podStartE2EDuration="2.555187631s" totalImagesPullingTime="404.997731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:28.553805374 +0000 UTC m=+799.617749335" watchObservedRunningTime="2026-09-17 18:16:28.555187631 +0000 UTC m=+799.619131593" Sep 17 18:16:28.717471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:28.717447 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7640-e893e6" Sep 17 18:16:28.807526 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:28.807433 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psqvt\" (UniqueName: \"kubernetes.io/projected/abaedb5a-7b12-4d68-b7b6-25011073ff44-kube-api-access-psqvt\") pod \"abaedb5a-7b12-4d68-b7b6-25011073ff44\" (UID: \"abaedb5a-7b12-4d68-b7b6-25011073ff44\") " Sep 17 18:16:28.809555 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:28.809523 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abaedb5a-7b12-4d68-b7b6-25011073ff44-kube-api-access-psqvt" (OuterVolumeSpecName: "kube-api-access-psqvt") pod "abaedb5a-7b12-4d68-b7b6-25011073ff44" (UID: "abaedb5a-7b12-4d68-b7b6-25011073ff44"). InnerVolumeSpecName "kube-api-access-psqvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:16:28.909246 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:28.909212 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psqvt\" (UniqueName: \"kubernetes.io/projected/abaedb5a-7b12-4d68-b7b6-25011073ff44-kube-api-access-psqvt\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:16:29.350179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:29.350139 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abaedb5a-7b12-4d68-b7b6-25011073ff44" path="/var/lib/kubelet/pods/abaedb5a-7b12-4d68-b7b6-25011073ff44/volumes" Sep 17 18:16:29.546201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:29.546163 2582 generic.go:356] "Generic (PLEG): container finished" podID="abaedb5a-7b12-4d68-b7b6-25011073ff44" containerID="773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7" exitCode=137 Sep 17 18:16:29.546201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:29.546214 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7640-e893e6" Sep 17 18:16:29.547275 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:29.546269 2582 scope.go:117] "RemoveContainer" containerID="773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7" Sep 17 18:16:29.555978 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:29.555953 2582 scope.go:117] "RemoveContainer" containerID="773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7" Sep 17 18:16:29.556306 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:29.556287 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7\": container with ID starting with 773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7 not found: ID does not exist" containerID="773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7" Sep 17 18:16:29.556367 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:29.556315 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7"} err="failed to get container status \"773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7\": rpc error: code = NotFound desc = could not find container \"773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7\": container with ID starting with 773c594a1c2ec38726c2ebb631a1d65d39c7a1c2c06fd24aee5fc17d5fcac4a7 not found: ID does not exist" Sep 17 18:16:31.524119 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:31.524093 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:16:32.532872 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:32.532845 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:16:34.903288 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:34.903255 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6"] Sep 17 18:16:34.907670 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:34.907646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:34.910079 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:34.910054 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-015dd779-serving-cert\"" Sep 17 18:16:34.922740 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:34.922713 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6"] Sep 17 18:16:35.073722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.073688 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.073929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.073751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw58f\" (UniqueName: \"kubernetes.io/projected/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-kube-api-access-hw58f\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.073929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.073855 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-openshift-service-ca\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.074070 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.073946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-metrics-certs\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.175345 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.175260 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-metrics-certs\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.175494 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.175429 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.175557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.175545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw58f\" (UniqueName: \"kubernetes.io/projected/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-kube-api-access-hw58f\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.175614 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:35.175564 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-serving-cert: secret "maas-api-e2e-ait-015dd779-serving-cert" not found Sep 17 18:16:35.175662 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.175613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-openshift-service-ca\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.175662 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:35.175645 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls podName:8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6 nodeName:}" failed. No retries permitted until 2026-09-17 18:16:35.675622171 +0000 UTC m=+806.739566126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls") pod "maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" (UID: "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6") : secret "maas-api-e2e-ait-015dd779-serving-cert" not found Sep 17 18:16:35.176162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.176145 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-openshift-service-ca\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.177813 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.177791 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-metrics-certs\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.183851 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.183831 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw58f\" (UniqueName: \"kubernetes.io/projected/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-kube-api-access-hw58f\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.560246 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.560163 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:16:35.680209 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.680175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.682432 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.682407 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls\") pod \"maas-api-e2e-ait-015dd779-7688b8f567-r7lj6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:35.826169 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:35.826088 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:36.107294 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:36.107266 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6"] Sep 17 18:16:36.109768 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:36.109740 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab3bc03_0a85_4b0d_ba7e_19b25592f7c6.slice/crio-e299aa065ae8a39e47366cc1d11207ae7316866582b6ef8dc3144cdf9fd48ce4 WatchSource:0}: Error finding container e299aa065ae8a39e47366cc1d11207ae7316866582b6ef8dc3144cdf9fd48ce4: Status 404 returned error can't find the container with id e299aa065ae8a39e47366cc1d11207ae7316866582b6ef8dc3144cdf9fd48ce4 Sep 17 18:16:36.594381 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:36.594353 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" event={"ID":"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6","Type":"ContainerStarted","Data":"e299aa065ae8a39e47366cc1d11207ae7316866582b6ef8dc3144cdf9fd48ce4"} Sep 17 18:16:37.529462 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:37.529432 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:16:37.603820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:37.603785 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" event={"ID":"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6","Type":"ContainerStarted","Data":"fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf"} Sep 17 18:16:37.604009 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:37.603901 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:37.640829 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:37.640759 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" podStartSLOduration=3.21408512 podStartE2EDuration="3.640738108s" totalImagesPullingTime="426.652988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:37.63730406 +0000 UTC m=+808.701248019" watchObservedRunningTime="2026-09-17 18:16:37.640738108 +0000 UTC m=+808.704682066" Sep 17 18:16:38.538428 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:38.538399 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:16:38.539011 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:38.538990 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:16:39.497000 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.496969 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8"] Sep 17 18:16:39.501063 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.501037 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.503486 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.503464 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-b200d5-serving-cert\"" Sep 17 18:16:39.511024 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.511004 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8"] Sep 17 18:16:39.620100 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.620073 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f129324b-abb2-4882-a824-a262b38c25db-openshift-service-ca\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.620471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.620109 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6499\" (UniqueName: \"kubernetes.io/projected/f129324b-abb2-4882-a824-a262b38c25db-kube-api-access-m6499\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.620471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.620159 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-metrics-certs\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.620471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.620247 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.720778 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.720748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.720965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.720797 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f129324b-abb2-4882-a824-a262b38c25db-openshift-service-ca\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.720965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.720828 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6499\" (UniqueName: \"kubernetes.io/projected/f129324b-abb2-4882-a824-a262b38c25db-kube-api-access-m6499\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.720965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.720896 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-metrics-certs\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.720965 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:39.720936 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-serving-cert: secret "maas-api-e2e-autores-b200d5-serving-cert" not found Sep 17 18:16:39.721135 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:39.721015 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls podName:f129324b-abb2-4882-a824-a262b38c25db nodeName:}" failed. No retries permitted until 2026-09-17 18:16:40.220992973 +0000 UTC m=+811.284936908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls") pod "maas-api-e2e-autores-b200d5-78547b494-62xv8" (UID: "f129324b-abb2-4882-a824-a262b38c25db") : secret "maas-api-e2e-autores-b200d5-serving-cert" not found Sep 17 18:16:39.721591 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.721563 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f129324b-abb2-4882-a824-a262b38c25db-openshift-service-ca\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.723557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.723536 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-metrics-certs\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:39.731233 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:39.731208 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6499\" (UniqueName: \"kubernetes.io/projected/f129324b-abb2-4882-a824-a262b38c25db-kube-api-access-m6499\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:40.227201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:40.227164 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:40.229604 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:40.229578 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls\") pod \"maas-api-e2e-autores-b200d5-78547b494-62xv8\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:40.413693 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:40.413659 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:40.689073 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:40.689049 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8"] Sep 17 18:16:40.690798 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:40.690775 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf129324b_abb2_4882_a824_a262b38c25db.slice/crio-9f3aa74aa44e665090def76b4f0ac627575eaa965afeae48f4a9b709709ab041 WatchSource:0}: Error finding container 9f3aa74aa44e665090def76b4f0ac627575eaa965afeae48f4a9b709709ab041: Status 404 returned error can't find the container with id 9f3aa74aa44e665090def76b4f0ac627575eaa965afeae48f4a9b709709ab041 Sep 17 18:16:41.629027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:41.628985 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" event={"ID":"f129324b-abb2-4882-a824-a262b38c25db","Type":"ContainerStarted","Data":"3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a"} Sep 17 18:16:41.629227 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:41.629068 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" event={"ID":"f129324b-abb2-4882-a824-a262b38c25db","Type":"ContainerStarted","Data":"9f3aa74aa44e665090def76b4f0ac627575eaa965afeae48f4a9b709709ab041"} Sep 17 18:16:41.629227 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:41.629103 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:41.649526 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:41.649478 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" podStartSLOduration=2.251665908 podStartE2EDuration="2.649462352s" totalImagesPullingTime="397.796444ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:41.646362263 +0000 UTC m=+812.710306223" watchObservedRunningTime="2026-09-17 18:16:41.649462352 +0000 UTC m=+812.713406310" Sep 17 18:16:43.616351 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:43.616318 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:16:45.909307 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:45.909272 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd"] Sep 17 18:16:45.913340 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:45.913318 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:45.916020 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:45.915956 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-99ac10-serving-cert\"" Sep 17 18:16:45.925490 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:45.925421 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd"] Sep 17 18:16:45.983184 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:45.983153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95n6k\" (UniqueName: \"kubernetes.io/projected/e46a520c-3553-4a0f-a997-e22b2ba0eda1-kube-api-access-95n6k\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:45.983350 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:45.983231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-metrics-certs\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:45.983405 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:45.983364 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:45.983509 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:45.983488 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46a520c-3553-4a0f-a997-e22b2ba0eda1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.084569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:46.084534 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46a520c-3553-4a0f-a997-e22b2ba0eda1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.084750 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:46.084604 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95n6k\" (UniqueName: \"kubernetes.io/projected/e46a520c-3553-4a0f-a997-e22b2ba0eda1-kube-api-access-95n6k\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.084750 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:46.084686 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-metrics-certs\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.084750 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:46.084739 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.084964 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:46.084941 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-serving-cert: secret "maas-api-e2e-inf-b-99ac10-serving-cert" not found Sep 17 18:16:46.085034 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:46.085024 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls podName:e46a520c-3553-4a0f-a997-e22b2ba0eda1 nodeName:}" failed. No retries permitted until 2026-09-17 18:16:46.5850023 +0000 UTC m=+817.648946243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls") pod "maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" (UID: "e46a520c-3553-4a0f-a997-e22b2ba0eda1") : secret "maas-api-e2e-inf-b-99ac10-serving-cert" not found Sep 17 18:16:46.085381 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:46.085359 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46a520c-3553-4a0f-a997-e22b2ba0eda1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.087277 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:46.087251 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-metrics-certs\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.093506 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:46.093484 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95n6k\" (UniqueName: \"kubernetes.io/projected/e46a520c-3553-4a0f-a997-e22b2ba0eda1-kube-api-access-95n6k\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.590037 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:46.590005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:46.590185 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:46.590155 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-serving-cert: secret "maas-api-e2e-inf-b-99ac10-serving-cert" not found Sep 17 18:16:46.590251 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:16:46.590241 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls podName:e46a520c-3553-4a0f-a997-e22b2ba0eda1 nodeName:}" failed. No retries permitted until 2026-09-17 18:16:47.590225388 +0000 UTC m=+818.654169324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls") pod "maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" (UID: "e46a520c-3553-4a0f-a997-e22b2ba0eda1") : secret "maas-api-e2e-inf-b-99ac10-serving-cert" not found Sep 17 18:16:47.601089 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:47.601052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:47.603488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:47.603465 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls\") pod \"maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:47.642216 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:47.642186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:16:47.729670 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:47.729637 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:48.036293 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:48.036256 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd"] Sep 17 18:16:48.040319 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:16:48.040288 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46a520c_3553_4a0f_a997_e22b2ba0eda1.slice/crio-c4efa8fe1989fdc2f1ee4fb6fa47bbd6b81205b85adb90a036cebc62690c2092 WatchSource:0}: Error finding container c4efa8fe1989fdc2f1ee4fb6fa47bbd6b81205b85adb90a036cebc62690c2092: Status 404 returned error can't find the container with id c4efa8fe1989fdc2f1ee4fb6fa47bbd6b81205b85adb90a036cebc62690c2092 Sep 17 18:16:48.675513 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:48.675427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" event={"ID":"e46a520c-3553-4a0f-a997-e22b2ba0eda1","Type":"ContainerStarted","Data":"97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7"} Sep 17 18:16:48.675513 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:48.675481 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" event={"ID":"e46a520c-3553-4a0f-a997-e22b2ba0eda1","Type":"ContainerStarted","Data":"c4efa8fe1989fdc2f1ee4fb6fa47bbd6b81205b85adb90a036cebc62690c2092"} Sep 17 18:16:48.675951 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:48.675529 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:16:48.693461 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:48.693414 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" podStartSLOduration=3.323574618 podStartE2EDuration="3.693399598s" totalImagesPullingTime="369.82498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:48.691700986 +0000 UTC m=+819.755644946" watchObservedRunningTime="2026-09-17 18:16:48.693399598 +0000 UTC m=+819.757343555" Sep 17 18:16:54.686818 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:16:54.686790 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:17:04.934398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:04.934345 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6"] Sep 17 18:17:04.934963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:04.934545 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" podUID="8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" containerName="maas-api" containerID="cri-o://fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf" gracePeriod=30 Sep 17 18:17:05.201561 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.201490 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:17:05.278979 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.278950 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-metrics-certs\") pod \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " Sep 17 18:17:05.279137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.279019 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls\") pod \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " Sep 17 18:17:05.279137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.279081 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-openshift-service-ca\") pod \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " Sep 17 18:17:05.279137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.279121 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw58f\" (UniqueName: \"kubernetes.io/projected/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-kube-api-access-hw58f\") pod \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\" (UID: \"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6\") " Sep 17 18:17:05.279504 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.279475 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" (UID: "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:17:05.281335 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.281304 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" (UID: "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:17:05.281425 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.281343 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" (UID: "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:17:05.281467 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.281455 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-kube-api-access-hw58f" (OuterVolumeSpecName: "kube-api-access-hw58f") pod "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" (UID: "8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6"). InnerVolumeSpecName "kube-api-access-hw58f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:17:05.380441 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.380407 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:17:05.380441 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.380436 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:17:05.380623 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.380448 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw58f\" (UniqueName: \"kubernetes.io/projected/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-kube-api-access-hw58f\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:17:05.380623 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.380456 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:17:05.777189 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.777149 2582 generic.go:356] "Generic (PLEG): container finished" podID="8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" containerID="fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf" exitCode=0 Sep 17 18:17:05.777411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.777234 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" Sep 17 18:17:05.777411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.777274 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" event={"ID":"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6","Type":"ContainerDied","Data":"fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf"} Sep 17 18:17:05.777411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.777356 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6" event={"ID":"8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6","Type":"ContainerDied","Data":"e299aa065ae8a39e47366cc1d11207ae7316866582b6ef8dc3144cdf9fd48ce4"} Sep 17 18:17:05.777411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.777389 2582 scope.go:117] "RemoveContainer" containerID="fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf" Sep 17 18:17:05.787381 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.787359 2582 scope.go:117] "RemoveContainer" containerID="fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf" Sep 17 18:17:05.787707 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:17:05.787680 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf\": container with ID starting with fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf not found: ID does not exist" containerID="fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf" Sep 17 18:17:05.787815 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.787714 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf"} err="failed to get container status \"fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf\": rpc error: code = NotFound desc = could not find container \"fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf\": container with ID starting with fc9427003ef930490961d7fe7094f4d4be7b99642699d1fdd351f75e8af019cf not found: ID does not exist" Sep 17 18:17:05.808023 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.807996 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6"] Sep 17 18:17:05.817453 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:05.817428 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-015dd779-7688b8f567-r7lj6"] Sep 17 18:17:07.347888 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:07.347835 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" path="/var/lib/kubelet/pods/8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6/volumes" Sep 17 18:17:10.923794 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:10.923763 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz"] Sep 17 18:17:10.924263 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:10.924250 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" containerName="maas-api" Sep 17 18:17:10.924263 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:10.924264 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" containerName="maas-api" Sep 17 18:17:10.924364 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:10.924355 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab3bc03-0a85-4b0d-ba7e-19b25592f7c6" containerName="maas-api" Sep 17 18:17:10.928936 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:10.928918 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:10.932139 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:10.932119 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4390f8ca-serving-cert\"" Sep 17 18:17:10.950378 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:10.950352 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz"] Sep 17 18:17:11.036699 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.036670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrq5j\" (UniqueName: \"kubernetes.io/projected/08b1711d-3481-4ee1-b82e-900771b2499c-kube-api-access-lrq5j\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.036915 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.036742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1711d-3481-4ee1-b82e-900771b2499c-openshift-service-ca\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.036915 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.036801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.036915 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.036885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-metrics-certs\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.138159 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.138120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrq5j\" (UniqueName: \"kubernetes.io/projected/08b1711d-3481-4ee1-b82e-900771b2499c-kube-api-access-lrq5j\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.138343 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.138226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1711d-3481-4ee1-b82e-900771b2499c-openshift-service-ca\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.138343 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.138282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.138343 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.138319 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-metrics-certs\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.138539 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:17:11.138483 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-serving-cert: secret "maas-api-e2e-migrate-4390f8ca-serving-cert" not found Sep 17 18:17:11.138589 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:17:11.138579 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls podName:08b1711d-3481-4ee1-b82e-900771b2499c nodeName:}" failed. No retries permitted until 2026-09-17 18:17:11.638557854 +0000 UTC m=+842.702501811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls") pod "maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" (UID: "08b1711d-3481-4ee1-b82e-900771b2499c") : secret "maas-api-e2e-migrate-4390f8ca-serving-cert" not found Sep 17 18:17:11.139124 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.139091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1711d-3481-4ee1-b82e-900771b2499c-openshift-service-ca\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.140963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.140940 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-metrics-certs\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.147334 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.147314 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrq5j\" (UniqueName: \"kubernetes.io/projected/08b1711d-3481-4ee1-b82e-900771b2499c-kube-api-access-lrq5j\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.644201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.644102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.646667 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.646642 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls\") pod \"maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:11.840260 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:11.840224 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:12.115769 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:12.115746 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz"] Sep 17 18:17:12.117675 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:17:12.117650 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b1711d_3481_4ee1_b82e_900771b2499c.slice/crio-ba8bee0ce6e119036a5cee6d6682127fb5f52f44f9e6780efb9b1670215e0d68 WatchSource:0}: Error finding container ba8bee0ce6e119036a5cee6d6682127fb5f52f44f9e6780efb9b1670215e0d68: Status 404 returned error can't find the container with id ba8bee0ce6e119036a5cee6d6682127fb5f52f44f9e6780efb9b1670215e0d68 Sep 17 18:17:12.819546 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:12.819469 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" event={"ID":"08b1711d-3481-4ee1-b82e-900771b2499c","Type":"ContainerStarted","Data":"b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b"} Sep 17 18:17:12.819546 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:12.819533 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" event={"ID":"08b1711d-3481-4ee1-b82e-900771b2499c","Type":"ContainerStarted","Data":"ba8bee0ce6e119036a5cee6d6682127fb5f52f44f9e6780efb9b1670215e0d68"} Sep 17 18:17:12.819721 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:12.819563 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:12.863503 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:12.863343 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" podStartSLOduration=2.472138452 podStartE2EDuration="2.863319958s" totalImagesPullingTime="391.181506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:12.859577127 +0000 UTC m=+843.923521109" watchObservedRunningTime="2026-09-17 18:17:12.863319958 +0000 UTC m=+843.927263918" Sep 17 18:17:18.830946 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:18.830919 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:39.560732 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.560694 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz"] Sep 17 18:17:39.561151 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.560970 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" podUID="08b1711d-3481-4ee1-b82e-900771b2499c" containerName="maas-api" containerID="cri-o://b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b" gracePeriod=30 Sep 17 18:17:39.831123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.831103 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:39.923845 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.923802 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls\") pod \"08b1711d-3481-4ee1-b82e-900771b2499c\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " Sep 17 18:17:39.924052 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.923921 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrq5j\" (UniqueName: \"kubernetes.io/projected/08b1711d-3481-4ee1-b82e-900771b2499c-kube-api-access-lrq5j\") pod \"08b1711d-3481-4ee1-b82e-900771b2499c\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " Sep 17 18:17:39.924052 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.923944 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-metrics-certs\") pod \"08b1711d-3481-4ee1-b82e-900771b2499c\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " Sep 17 18:17:39.924052 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.923963 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1711d-3481-4ee1-b82e-900771b2499c-openshift-service-ca\") pod \"08b1711d-3481-4ee1-b82e-900771b2499c\" (UID: \"08b1711d-3481-4ee1-b82e-900771b2499c\") " Sep 17 18:17:39.924392 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.924362 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b1711d-3481-4ee1-b82e-900771b2499c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08b1711d-3481-4ee1-b82e-900771b2499c" (UID: "08b1711d-3481-4ee1-b82e-900771b2499c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:17:39.927030 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.926987 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08b1711d-3481-4ee1-b82e-900771b2499c" (UID: "08b1711d-3481-4ee1-b82e-900771b2499c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:17:39.928296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.928270 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b1711d-3481-4ee1-b82e-900771b2499c-kube-api-access-lrq5j" (OuterVolumeSpecName: "kube-api-access-lrq5j") pod "08b1711d-3481-4ee1-b82e-900771b2499c" (UID: "08b1711d-3481-4ee1-b82e-900771b2499c"). InnerVolumeSpecName "kube-api-access-lrq5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:17:39.929666 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.929641 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "08b1711d-3481-4ee1-b82e-900771b2499c" (UID: "08b1711d-3481-4ee1-b82e-900771b2499c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:17:39.994826 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.994772 2582 generic.go:356] "Generic (PLEG): container finished" podID="08b1711d-3481-4ee1-b82e-900771b2499c" containerID="b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b" exitCode=0 Sep 17 18:17:39.994826 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.994823 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" Sep 17 18:17:39.995069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.994874 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" event={"ID":"08b1711d-3481-4ee1-b82e-900771b2499c","Type":"ContainerDied","Data":"b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b"} Sep 17 18:17:39.995069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.994929 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz" event={"ID":"08b1711d-3481-4ee1-b82e-900771b2499c","Type":"ContainerDied","Data":"ba8bee0ce6e119036a5cee6d6682127fb5f52f44f9e6780efb9b1670215e0d68"} Sep 17 18:17:39.995069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:39.994951 2582 scope.go:117] "RemoveContainer" containerID="b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b" Sep 17 18:17:40.005416 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:40.005399 2582 scope.go:117] "RemoveContainer" containerID="b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b" Sep 17 18:17:40.005690 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:17:40.005668 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b\": container with ID starting with b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b not found: ID does not exist" containerID="b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b" Sep 17 18:17:40.005745 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:40.005698 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b"} err="failed to get container status \"b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b\": rpc error: code = NotFound desc = could not find container \"b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b\": container with ID starting with b45475873c313dfb570fba130aeed495c87cfb06f6ae528b46ce792f7210ab7b not found: ID does not exist" Sep 17 18:17:40.017761 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:40.017740 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz"] Sep 17 18:17:40.021021 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:40.021000 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4390f8ca-5df874f57-v4gsz"] Sep 17 18:17:40.025090 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:40.025073 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:17:40.025158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:40.025092 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrq5j\" (UniqueName: \"kubernetes.io/projected/08b1711d-3481-4ee1-b82e-900771b2499c-kube-api-access-lrq5j\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:17:40.025158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:40.025104 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08b1711d-3481-4ee1-b82e-900771b2499c-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:17:40.025158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:40.025114 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b1711d-3481-4ee1-b82e-900771b2499c-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:17:41.348411 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:41.348379 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b1711d-3481-4ee1-b82e-900771b2499c" path="/var/lib/kubelet/pods/08b1711d-3481-4ee1-b82e-900771b2499c/volumes" Sep 17 18:17:45.116929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.116849 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm"] Sep 17 18:17:45.117506 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.117485 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b1711d-3481-4ee1-b82e-900771b2499c" containerName="maas-api" Sep 17 18:17:45.117506 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.117507 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b1711d-3481-4ee1-b82e-900771b2499c" containerName="maas-api" Sep 17 18:17:45.117667 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.117574 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b1711d-3481-4ee1-b82e-900771b2499c" containerName="maas-api" Sep 17 18:17:45.122051 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.122028 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.124773 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.124753 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7fd35bf2-serving-cert\"" Sep 17 18:17:45.132335 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.132309 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm"] Sep 17 18:17:45.280684 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.280646 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.280853 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.280731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-openshift-service-ca\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.280853 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.280780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-metrics-certs\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.280956 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.280885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b5vq\" (UniqueName: \"kubernetes.io/projected/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-kube-api-access-8b5vq\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.381905 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.381808 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.382049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.381906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-openshift-service-ca\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.382049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.381971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-metrics-certs\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.382049 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:17:45.381985 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-serving-cert: secret "maas-api-e2e-ait-7fd35bf2-serving-cert" not found Sep 17 18:17:45.382049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.382024 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b5vq\" (UniqueName: \"kubernetes.io/projected/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-kube-api-access-8b5vq\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.382049 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:17:45.382045 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls podName:2da529bd-8a80-4b2f-8df7-42a27b3c42b2 nodeName:}" failed. No retries permitted until 2026-09-17 18:17:45.882025613 +0000 UTC m=+876.945969563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls") pod "maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" (UID: "2da529bd-8a80-4b2f-8df7-42a27b3c42b2") : secret "maas-api-e2e-ait-7fd35bf2-serving-cert" not found Sep 17 18:17:45.382577 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.382548 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-openshift-service-ca\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.384372 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.384333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-metrics-certs\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.392197 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.392168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b5vq\" (UniqueName: \"kubernetes.io/projected/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-kube-api-access-8b5vq\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.886964 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.886928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:45.889536 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:45.889506 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls\") pod \"maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:46.036869 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:46.036827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:46.313732 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:46.313708 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm"] Sep 17 18:17:46.315613 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:17:46.315590 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da529bd_8a80_4b2f_8df7_42a27b3c42b2.slice/crio-1d32819034d3f59ad2bf76b655ea3018f516ea1bd48c0ba690e754c27123b147 WatchSource:0}: Error finding container 1d32819034d3f59ad2bf76b655ea3018f516ea1bd48c0ba690e754c27123b147: Status 404 returned error can't find the container with id 1d32819034d3f59ad2bf76b655ea3018f516ea1bd48c0ba690e754c27123b147 Sep 17 18:17:47.038136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:47.038042 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" event={"ID":"2da529bd-8a80-4b2f-8df7-42a27b3c42b2","Type":"ContainerStarted","Data":"b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d"} Sep 17 18:17:47.038136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:47.038110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" event={"ID":"2da529bd-8a80-4b2f-8df7-42a27b3c42b2","Type":"ContainerStarted","Data":"1d32819034d3f59ad2bf76b655ea3018f516ea1bd48c0ba690e754c27123b147"} Sep 17 18:17:47.038356 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:47.038151 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:17:47.056352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:47.056293 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" podStartSLOduration=1.6013053350000002 podStartE2EDuration="2.056273511s" totalImagesPullingTime="454.968176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:47.055690842 +0000 UTC m=+878.119634801" watchObservedRunningTime="2026-09-17 18:17:47.056273511 +0000 UTC m=+878.120217473" Sep 17 18:17:53.051074 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:17:53.051041 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:18:11.287226 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:11.287193 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:18:11.290534 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:11.290512 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:18:11.306629 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:11.306598 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:18:11.309995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:11.309977 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:18:14.370320 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.370278 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm"] Sep 17 18:18:14.371162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.371101 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" podUID="2da529bd-8a80-4b2f-8df7-42a27b3c42b2" containerName="maas-api" containerID="cri-o://b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d" gracePeriod=30 Sep 17 18:18:14.657568 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.657542 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:18:14.781749 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.781715 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-metrics-certs\") pod \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " Sep 17 18:18:14.781959 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.781775 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-openshift-service-ca\") pod \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " Sep 17 18:18:14.781959 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.781814 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b5vq\" (UniqueName: \"kubernetes.io/projected/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-kube-api-access-8b5vq\") pod \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " Sep 17 18:18:14.781959 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.781897 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls\") pod \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\" (UID: \"2da529bd-8a80-4b2f-8df7-42a27b3c42b2\") " Sep 17 18:18:14.782196 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.782174 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2da529bd-8a80-4b2f-8df7-42a27b3c42b2" (UID: "2da529bd-8a80-4b2f-8df7-42a27b3c42b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:18:14.782365 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.782349 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:18:14.784043 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.784016 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2da529bd-8a80-4b2f-8df7-42a27b3c42b2" (UID: "2da529bd-8a80-4b2f-8df7-42a27b3c42b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:18:14.784155 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.784037 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2da529bd-8a80-4b2f-8df7-42a27b3c42b2" (UID: "2da529bd-8a80-4b2f-8df7-42a27b3c42b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:18:14.784155 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.784088 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-kube-api-access-8b5vq" (OuterVolumeSpecName: "kube-api-access-8b5vq") pod "2da529bd-8a80-4b2f-8df7-42a27b3c42b2" (UID: "2da529bd-8a80-4b2f-8df7-42a27b3c42b2"). InnerVolumeSpecName "kube-api-access-8b5vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:18:14.883490 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.883416 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:18:14.883490 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.883441 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b5vq\" (UniqueName: \"kubernetes.io/projected/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-kube-api-access-8b5vq\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:18:14.883490 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.883450 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da529bd-8a80-4b2f-8df7-42a27b3c42b2-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:18:14.883691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.883522 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb"] Sep 17 18:18:14.884112 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.884098 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da529bd-8a80-4b2f-8df7-42a27b3c42b2" containerName="maas-api" Sep 17 18:18:14.884154 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.884114 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da529bd-8a80-4b2f-8df7-42a27b3c42b2" containerName="maas-api" Sep 17 18:18:14.884202 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.884193 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da529bd-8a80-4b2f-8df7-42a27b3c42b2" containerName="maas-api" Sep 17 18:18:14.887505 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.887490 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:14.890047 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.890029 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b200d5\"/\"openshift-service-ca.crt\"" Sep 17 18:18:14.890814 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.890800 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b200d5\"/\"default-dockercfg-xfs5h\"" Sep 17 18:18:14.890916 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.890840 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b200d5\"/\"autores-model-b200d5-kserve-self-signed-certs\"" Sep 17 18:18:14.890972 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.890843 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b200d5\"/\"kube-root-ca.crt\"" Sep 17 18:18:14.896150 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.896116 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb"] Sep 17 18:18:14.985062 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.985025 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-home\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:14.985237 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.985070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tls-certs\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:14.985237 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.985147 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-dshm\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:14.985237 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.985215 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tmp-dir\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:14.985344 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.985276 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-model-cache\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:14.985379 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:14.985343 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7b5\" (UniqueName: \"kubernetes.io/projected/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-kube-api-access-9q7b5\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.086426 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.086393 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7b5\" (UniqueName: \"kubernetes.io/projected/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-kube-api-access-9q7b5\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.086601 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.086435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-home\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.086601 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.086551 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tls-certs\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.086689 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.086626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-dshm\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.086689 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.086673 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tmp-dir\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.086817 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.086781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-home\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.086817 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.086805 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-model-cache\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.087100 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.087055 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tmp-dir\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.087100 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.087088 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-model-cache\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.088910 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.088886 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-dshm\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.089264 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.089247 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tls-certs\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.094098 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.094075 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7b5\" (UniqueName: \"kubernetes.io/projected/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-kube-api-access-9q7b5\") pod \"autores-model-b200d5-kserve-856ff49947-krcqb\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.199909 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.199878 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:15.203237 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.203209 2582 generic.go:356] "Generic (PLEG): container finished" podID="2da529bd-8a80-4b2f-8df7-42a27b3c42b2" containerID="b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d" exitCode=0 Sep 17 18:18:15.203326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.203256 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" Sep 17 18:18:15.203326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.203287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" event={"ID":"2da529bd-8a80-4b2f-8df7-42a27b3c42b2","Type":"ContainerDied","Data":"b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d"} Sep 17 18:18:15.203448 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.203345 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm" event={"ID":"2da529bd-8a80-4b2f-8df7-42a27b3c42b2","Type":"ContainerDied","Data":"1d32819034d3f59ad2bf76b655ea3018f516ea1bd48c0ba690e754c27123b147"} Sep 17 18:18:15.203448 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.203373 2582 scope.go:117] "RemoveContainer" containerID="b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d" Sep 17 18:18:15.212736 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.212710 2582 scope.go:117] "RemoveContainer" containerID="b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d" Sep 17 18:18:15.213046 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:18:15.213020 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d\": container with ID starting with b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d not found: ID does not exist" containerID="b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d" Sep 17 18:18:15.213145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.213049 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d"} err="failed to get container status \"b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d\": rpc error: code = NotFound desc = could not find container \"b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d\": container with ID starting with b09e9c30a376f465d5728a5ae5cbe88616f0e80e2f3b9a5ddfbf7e06679f3e3d not found: ID does not exist" Sep 17 18:18:15.229093 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.229067 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm"] Sep 17 18:18:15.232502 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.232480 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fd35bf2-589cb45c9-82jkm"] Sep 17 18:18:15.348688 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.348651 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2da529bd-8a80-4b2f-8df7-42a27b3c42b2" path="/var/lib/kubelet/pods/2da529bd-8a80-4b2f-8df7-42a27b3c42b2/volumes" Sep 17 18:18:15.493699 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:15.493673 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb"] Sep 17 18:18:15.495933 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:18:15.495909 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c2206f7_d5fe_472b_9083_d8e2ccbb09f0.slice/crio-17c2a82741c3709b0703fc44800ef3e4eb6f0183e9a71e98fbf344e266a75e23 WatchSource:0}: Error finding container 17c2a82741c3709b0703fc44800ef3e4eb6f0183e9a71e98fbf344e266a75e23: Status 404 returned error can't find the container with id 17c2a82741c3709b0703fc44800ef3e4eb6f0183e9a71e98fbf344e266a75e23 Sep 17 18:18:16.210704 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:16.210665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" event={"ID":"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0","Type":"ContainerStarted","Data":"44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2"} Sep 17 18:18:16.210949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:16.210719 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" event={"ID":"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0","Type":"ContainerStarted","Data":"17c2a82741c3709b0703fc44800ef3e4eb6f0183e9a71e98fbf344e266a75e23"} Sep 17 18:18:23.325602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.325547 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" podStartSLOduration=9.325525872 podStartE2EDuration="9.325525872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:16.230292041 +0000 UTC m=+907.294236000" watchObservedRunningTime="2026-09-17 18:18:23.325525872 +0000 UTC m=+914.389469833" Sep 17 18:18:23.327401 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.327373 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z"] Sep 17 18:18:23.331469 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.331450 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.333687 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.333667 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-700d65b8-serving-cert\"" Sep 17 18:18:23.353776 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.353740 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z"] Sep 17 18:18:23.478486 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.478449 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-openshift-service-ca\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.478679 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.478501 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-maas-api-tls\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.478679 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.478613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd66z\" (UniqueName: \"kubernetes.io/projected/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-kube-api-access-zd66z\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.478679 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.478670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-metrics-certs\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.580018 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.579917 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-openshift-service-ca\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.580018 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.579977 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-maas-api-tls\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.580234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.580039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd66z\" (UniqueName: \"kubernetes.io/projected/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-kube-api-access-zd66z\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.580234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.580081 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-metrics-certs\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.580728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.580705 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-openshift-service-ca\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.582639 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.582615 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-metrics-certs\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.582738 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.582666 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-maas-api-tls\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.588230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.588205 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd66z\" (UniqueName: \"kubernetes.io/projected/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-kube-api-access-zd66z\") pod \"maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.650307 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.650270 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:23.938614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:23.938474 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z"] Sep 17 18:18:23.942070 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:18:23.942019 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fcb7b5_89f5_4b48_ab94_ea3ac9319f5e.slice/crio-421feafdfde48036061f5733129ffe1d1f7e6083c3b5f38631e3956fc98b7d3a WatchSource:0}: Error finding container 421feafdfde48036061f5733129ffe1d1f7e6083c3b5f38631e3956fc98b7d3a: Status 404 returned error can't find the container with id 421feafdfde48036061f5733129ffe1d1f7e6083c3b5f38631e3956fc98b7d3a Sep 17 18:18:24.259927 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:24.259813 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" event={"ID":"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e","Type":"ContainerStarted","Data":"421feafdfde48036061f5733129ffe1d1f7e6083c3b5f38631e3956fc98b7d3a"} Sep 17 18:18:25.200129 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:25.200096 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:25.200513 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:25.200146 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:25.212948 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:25.212918 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:25.267169 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:25.267131 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" event={"ID":"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e","Type":"ContainerStarted","Data":"132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930"} Sep 17 18:18:25.267499 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:25.267470 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:25.278548 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:25.278521 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:18:25.286667 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:25.286619 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" podStartSLOduration=1.899847801 podStartE2EDuration="2.286605122s" totalImagesPullingTime="386.757321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:25.284052049 +0000 UTC m=+916.347996008" watchObservedRunningTime="2026-09-17 18:18:25.286605122 +0000 UTC m=+916.350549078" Sep 17 18:18:31.278978 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:31.278946 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:18:59.823084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:59.823044 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8"] Sep 17 18:18:59.823604 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:18:59.823302 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" podUID="f129324b-abb2-4882-a824-a262b38c25db" containerName="maas-api" containerID="cri-o://3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a" gracePeriod=30 Sep 17 18:19:00.080771 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.080705 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:19:00.256245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.256209 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f129324b-abb2-4882-a824-a262b38c25db-openshift-service-ca\") pod \"f129324b-abb2-4882-a824-a262b38c25db\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " Sep 17 18:19:00.256443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.256272 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6499\" (UniqueName: \"kubernetes.io/projected/f129324b-abb2-4882-a824-a262b38c25db-kube-api-access-m6499\") pod \"f129324b-abb2-4882-a824-a262b38c25db\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " Sep 17 18:19:00.256443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.256374 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls\") pod \"f129324b-abb2-4882-a824-a262b38c25db\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " Sep 17 18:19:00.256605 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.256533 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-metrics-certs\") pod \"f129324b-abb2-4882-a824-a262b38c25db\" (UID: \"f129324b-abb2-4882-a824-a262b38c25db\") " Sep 17 18:19:00.256665 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.256622 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f129324b-abb2-4882-a824-a262b38c25db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f129324b-abb2-4882-a824-a262b38c25db" (UID: "f129324b-abb2-4882-a824-a262b38c25db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:19:00.256998 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.256974 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f129324b-abb2-4882-a824-a262b38c25db-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:00.258402 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.258374 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f129324b-abb2-4882-a824-a262b38c25db-kube-api-access-m6499" (OuterVolumeSpecName: "kube-api-access-m6499") pod "f129324b-abb2-4882-a824-a262b38c25db" (UID: "f129324b-abb2-4882-a824-a262b38c25db"). InnerVolumeSpecName "kube-api-access-m6499". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:19:00.258564 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.258547 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f129324b-abb2-4882-a824-a262b38c25db" (UID: "f129324b-abb2-4882-a824-a262b38c25db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:19:00.258995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.258970 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f129324b-abb2-4882-a824-a262b38c25db" (UID: "f129324b-abb2-4882-a824-a262b38c25db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:19:00.357793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.357703 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:00.357793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.357730 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6499\" (UniqueName: \"kubernetes.io/projected/f129324b-abb2-4882-a824-a262b38c25db-kube-api-access-m6499\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:00.357793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.357741 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f129324b-abb2-4882-a824-a262b38c25db-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:00.472378 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.472343 2582 generic.go:356] "Generic (PLEG): container finished" podID="f129324b-abb2-4882-a824-a262b38c25db" containerID="3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a" exitCode=0 Sep 17 18:19:00.472556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.472386 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" event={"ID":"f129324b-abb2-4882-a824-a262b38c25db","Type":"ContainerDied","Data":"3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a"} Sep 17 18:19:00.472556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.472454 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" event={"ID":"f129324b-abb2-4882-a824-a262b38c25db","Type":"ContainerDied","Data":"9f3aa74aa44e665090def76b4f0ac627575eaa965afeae48f4a9b709709ab041"} Sep 17 18:19:00.472556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.472400 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8" Sep 17 18:19:00.472690 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.472483 2582 scope.go:117] "RemoveContainer" containerID="3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a" Sep 17 18:19:00.482995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.482976 2582 scope.go:117] "RemoveContainer" containerID="3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a" Sep 17 18:19:00.483278 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:19:00.483259 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a\": container with ID starting with 3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a not found: ID does not exist" containerID="3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a" Sep 17 18:19:00.483330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.483290 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a"} err="failed to get container status \"3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a\": rpc error: code = NotFound desc = could not find container \"3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a\": container with ID starting with 3269c49cb3f3e70f1c09ce536478aafe39cf58b727f4230235c5339f86a9304a not found: ID does not exist" Sep 17 18:19:00.495413 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.495389 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8"] Sep 17 18:19:00.497742 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:00.497717 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b200d5-78547b494-62xv8"] Sep 17 18:19:01.348829 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:01.348799 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f129324b-abb2-4882-a824-a262b38c25db" path="/var/lib/kubelet/pods/f129324b-abb2-4882-a824-a262b38c25db/volumes" Sep 17 18:19:04.396726 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.396687 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z"] Sep 17 18:19:04.397422 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.397371 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" podUID="94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" containerName="maas-api" containerID="cri-o://132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930" gracePeriod=30 Sep 17 18:19:04.665576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.665553 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:19:04.702393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.702361 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd66z\" (UniqueName: \"kubernetes.io/projected/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-kube-api-access-zd66z\") pod \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " Sep 17 18:19:04.702573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.702426 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-metrics-certs\") pod \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " Sep 17 18:19:04.702573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.702462 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-openshift-service-ca\") pod \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " Sep 17 18:19:04.702573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.702563 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-maas-api-tls\") pod \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\" (UID: \"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e\") " Sep 17 18:19:04.702933 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.702905 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" (UID: "94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:19:04.704704 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.704667 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" (UID: "94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:19:04.705062 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.705037 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-kube-api-access-zd66z" (OuterVolumeSpecName: "kube-api-access-zd66z") pod "94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" (UID: "94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e"). InnerVolumeSpecName "kube-api-access-zd66z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:19:04.705236 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.705202 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" (UID: "94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:19:04.804133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.804102 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd66z\" (UniqueName: \"kubernetes.io/projected/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-kube-api-access-zd66z\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:04.804133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.804129 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:04.804322 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.804143 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:04.804322 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:04.804155 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:05.507420 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.507391 2582 generic.go:356] "Generic (PLEG): container finished" podID="94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" containerID="132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930" exitCode=0 Sep 17 18:19:05.507819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.507449 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" Sep 17 18:19:05.507819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.507462 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" event={"ID":"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e","Type":"ContainerDied","Data":"132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930"} Sep 17 18:19:05.507819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.507502 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z" event={"ID":"94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e","Type":"ContainerDied","Data":"421feafdfde48036061f5733129ffe1d1f7e6083c3b5f38631e3956fc98b7d3a"} Sep 17 18:19:05.507819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.507523 2582 scope.go:117] "RemoveContainer" containerID="132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930" Sep 17 18:19:05.507819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.507766 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb"] Sep 17 18:19:05.517527 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.517511 2582 scope.go:117] "RemoveContainer" containerID="132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930" Sep 17 18:19:05.517790 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:19:05.517771 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930\": container with ID starting with 132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930 not found: ID does not exist" containerID="132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930" Sep 17 18:19:05.517841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.517798 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930"} err="failed to get container status \"132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930\": rpc error: code = NotFound desc = could not find container \"132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930\": container with ID starting with 132e360b3dcecd724a93f4300cfbd257605105c2e7ee3b062173e18bfb70f930 not found: ID does not exist" Sep 17 18:19:05.551365 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.551334 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z"] Sep 17 18:19:05.558493 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:05.558463 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-700d65b8-fccff6b8f-wmt8z"] Sep 17 18:19:07.347200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:07.347165 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" path="/var/lib/kubelet/pods/94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e/volumes" Sep 17 18:19:20.527994 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.527946 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" podUID="5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" containerName="main" containerID="cri-o://44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2" gracePeriod=45 Sep 17 18:19:20.791316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.791250 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:19:20.867170 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867139 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-model-cache\") pod \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " Sep 17 18:19:20.867347 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867178 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tls-certs\") pod \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " Sep 17 18:19:20.867347 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867204 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q7b5\" (UniqueName: \"kubernetes.io/projected/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-kube-api-access-9q7b5\") pod \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " Sep 17 18:19:20.867347 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867252 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-dshm\") pod \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " Sep 17 18:19:20.867520 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867379 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tmp-dir\") pod \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " Sep 17 18:19:20.867520 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867408 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-home\") pod \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\" (UID: \"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0\") " Sep 17 18:19:20.867520 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867469 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" (UID: "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:20.867688 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867656 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" (UID: "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:20.867814 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867791 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-home" (OuterVolumeSpecName: "home") pod "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" (UID: "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:20.867936 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.867913 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:20.868092 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.868077 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:20.868192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.868180 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:20.869594 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.869566 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-dshm" (OuterVolumeSpecName: "dshm") pod "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" (UID: "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:19:20.869698 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.869642 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-kube-api-access-9q7b5" (OuterVolumeSpecName: "kube-api-access-9q7b5") pod "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" (UID: "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0"). InnerVolumeSpecName "kube-api-access-9q7b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:19:20.869983 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.869958 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" (UID: "5c2206f7-d5fe-472b-9083-d8e2ccbb09f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:19:20.969341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.969308 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:20.969341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.969336 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q7b5\" (UniqueName: \"kubernetes.io/projected/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-kube-api-access-9q7b5\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:20.969341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:20.969346 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:19:21.605316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.605229 2582 generic.go:356] "Generic (PLEG): container finished" podID="5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" containerID="44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2" exitCode=0 Sep 17 18:19:21.605316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.605275 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" Sep 17 18:19:21.605316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.605284 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" event={"ID":"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0","Type":"ContainerDied","Data":"44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2"} Sep 17 18:19:21.605819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.605334 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb" event={"ID":"5c2206f7-d5fe-472b-9083-d8e2ccbb09f0","Type":"ContainerDied","Data":"17c2a82741c3709b0703fc44800ef3e4eb6f0183e9a71e98fbf344e266a75e23"} Sep 17 18:19:21.605819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.605357 2582 scope.go:117] "RemoveContainer" containerID="44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2" Sep 17 18:19:21.614542 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.614527 2582 scope.go:117] "RemoveContainer" containerID="44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2" Sep 17 18:19:21.614792 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:19:21.614775 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2\": container with ID starting with 44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2 not found: ID does not exist" containerID="44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2" Sep 17 18:19:21.614850 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.614801 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2"} err="failed to get container status \"44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2\": rpc error: code = NotFound desc = could not find container \"44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2\": container with ID starting with 44c34ab12c29000d86183aad505fd134837a7f08ad3b468e1e0a52bbd17371c2 not found: ID does not exist" Sep 17 18:19:21.622373 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.622350 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb"] Sep 17 18:19:21.624458 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:21.624441 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b200d5/autores-model-b200d5-kserve-856ff49947-krcqb"] Sep 17 18:19:23.346988 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:23.346913 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" path="/var/lib/kubelet/pods/5c2206f7-d5fe-472b-9083-d8e2ccbb09f0/volumes" Sep 17 18:19:43.088628 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.088595 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p"] Sep 17 18:19:43.089391 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089358 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f129324b-abb2-4882-a824-a262b38c25db" containerName="maas-api" Sep 17 18:19:43.089391 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089385 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f129324b-abb2-4882-a824-a262b38c25db" containerName="maas-api" Sep 17 18:19:43.089599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089406 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" containerName="maas-api" Sep 17 18:19:43.089599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089414 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" containerName="maas-api" Sep 17 18:19:43.089599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089429 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" containerName="main" Sep 17 18:19:43.089599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089437 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" containerName="main" Sep 17 18:19:43.089599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089547 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="94fcb7b5-89f5-4b48-ab94-ea3ac9319f5e" containerName="maas-api" Sep 17 18:19:43.089599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089566 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c2206f7-d5fe-472b-9083-d8e2ccbb09f0" containerName="main" Sep 17 18:19:43.089599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.089577 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f129324b-abb2-4882-a824-a262b38c25db" containerName="maas-api" Sep 17 18:19:43.092980 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.092960 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.095404 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.095385 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"openshift-service-ca.crt\"" Sep 17 18:19:43.096457 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.096441 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"premium-w4-b18433-kserve-self-signed-certs\"" Sep 17 18:19:43.096518 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.096442 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"default-dockercfg-2nzcz\"" Sep 17 18:19:43.096518 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.096477 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"kube-root-ca.crt\"" Sep 17 18:19:43.103428 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.103403 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p"] Sep 17 18:19:43.184218 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.184184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655p2\" (UniqueName: \"kubernetes.io/projected/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-kube-api-access-655p2\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.184389 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.184261 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-model-cache\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.184389 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.184323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-home\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.184478 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.184466 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tmp-dir\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.184526 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.184494 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tls-certs\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.184573 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.184532 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-dshm\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.285927 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.285883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tmp-dir\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.285927 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.285927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tls-certs\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.286182 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.285970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-dshm\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.286182 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.286009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655p2\" (UniqueName: \"kubernetes.io/projected/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-kube-api-access-655p2\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.286182 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.286066 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-model-cache\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.286182 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.286099 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-home\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.286409 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.286377 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tmp-dir\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.286479 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.286419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-model-cache\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.286536 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.286490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-home\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.288215 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.288190 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-dshm\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.288465 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.288450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tls-certs\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.294482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.294466 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-655p2\" (UniqueName: \"kubernetes.io/projected/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-kube-api-access-655p2\") pod \"premium-w4-b18433-kserve-69fdf5bc8f-tjr2p\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.402570 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.402545 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:43.688659 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.688621 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p"] Sep 17 18:19:43.690692 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:19:43.690661 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6a795bf_6f73_494e_8bb8_bd9e23407bf5.slice/crio-d2873cab3a76facbf786e8cd188fdf633010b14f13ea046d87edd9156ace1c3b WatchSource:0}: Error finding container d2873cab3a76facbf786e8cd188fdf633010b14f13ea046d87edd9156ace1c3b: Status 404 returned error can't find the container with id d2873cab3a76facbf786e8cd188fdf633010b14f13ea046d87edd9156ace1c3b Sep 17 18:19:43.729342 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:43.729316 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" event={"ID":"c6a795bf-6f73-494e-8bb8-bd9e23407bf5","Type":"ContainerStarted","Data":"d2873cab3a76facbf786e8cd188fdf633010b14f13ea046d87edd9156ace1c3b"} Sep 17 18:19:44.736625 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:44.736582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" event={"ID":"c6a795bf-6f73-494e-8bb8-bd9e23407bf5","Type":"ContainerStarted","Data":"b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab"} Sep 17 18:19:44.755485 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:44.755438 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" podStartSLOduration=1.7554234229999999 podStartE2EDuration="1.755423423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:44.752607491 +0000 UTC m=+995.816551450" watchObservedRunningTime="2026-09-17 18:19:44.755423423 +0000 UTC m=+995.819367380" Sep 17 18:19:50.035529 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.035494 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d"] Sep 17 18:19:50.039985 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.039960 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.043964 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.043207 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-83507f-serving-cert\"" Sep 17 18:19:50.055512 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.055479 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d"] Sep 17 18:19:50.055994 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.055971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p958s\" (UniqueName: \"kubernetes.io/projected/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-kube-api-access-p958s\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.056049 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.056030 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.056197 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.056169 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-metrics-certs\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.056330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.056265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-openshift-service-ca\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.157571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.157538 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-openshift-service-ca\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.157741 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.157592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p958s\" (UniqueName: \"kubernetes.io/projected/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-kube-api-access-p958s\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.157741 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.157616 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.157741 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.157668 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-metrics-certs\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.157934 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:19:50.157761 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-serving-cert: secret "maas-api-e2e-explicit-83507f-serving-cert" not found Sep 17 18:19:50.157934 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:19:50.157832 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls podName:1db9b14b-80cf-4fbf-9054-4092c8b87e6c nodeName:}" failed. No retries permitted until 2026-09-17 18:19:50.657809847 +0000 UTC m=+1001.721753789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls") pod "maas-api-e2e-explicit-83507f-5c57547c49-phv5d" (UID: "1db9b14b-80cf-4fbf-9054-4092c8b87e6c") : secret "maas-api-e2e-explicit-83507f-serving-cert" not found Sep 17 18:19:50.158302 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.158279 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-openshift-service-ca\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.160275 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.160252 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-metrics-certs\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.167043 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.167024 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p958s\" (UniqueName: \"kubernetes.io/projected/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-kube-api-access-p958s\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.663149 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:50.662559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:50.663149 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:19:50.662938 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-serving-cert: secret "maas-api-e2e-explicit-83507f-serving-cert" not found Sep 17 18:19:50.663149 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:19:50.663010 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls podName:1db9b14b-80cf-4fbf-9054-4092c8b87e6c nodeName:}" failed. No retries permitted until 2026-09-17 18:19:51.662989455 +0000 UTC m=+1002.726933391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls") pod "maas-api-e2e-explicit-83507f-5c57547c49-phv5d" (UID: "1db9b14b-80cf-4fbf-9054-4092c8b87e6c") : secret "maas-api-e2e-explicit-83507f-serving-cert" not found Sep 17 18:19:51.674681 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:51.674647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:51.677164 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:51.677140 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls\") pod \"maas-api-e2e-explicit-83507f-5c57547c49-phv5d\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:51.861975 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:51.861938 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:52.135360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:52.135333 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d"] Sep 17 18:19:52.137671 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:19:52.137646 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db9b14b_80cf_4fbf_9054_4092c8b87e6c.slice/crio-66bdd0b72fab21929fab853c8f6765fd73b5ff866149ef014b4f7d89080aeb7b WatchSource:0}: Error finding container 66bdd0b72fab21929fab853c8f6765fd73b5ff866149ef014b4f7d89080aeb7b: Status 404 returned error can't find the container with id 66bdd0b72fab21929fab853c8f6765fd73b5ff866149ef014b4f7d89080aeb7b Sep 17 18:19:52.139189 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:52.139168 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:19:52.784985 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:52.784884 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" event={"ID":"1db9b14b-80cf-4fbf-9054-4092c8b87e6c","Type":"ContainerStarted","Data":"5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f"} Sep 17 18:19:52.784985 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:52.784951 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" event={"ID":"1db9b14b-80cf-4fbf-9054-4092c8b87e6c","Type":"ContainerStarted","Data":"66bdd0b72fab21929fab853c8f6765fd73b5ff866149ef014b4f7d89080aeb7b"} Sep 17 18:19:52.785384 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:52.785062 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:19:52.804634 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:52.804582 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" podStartSLOduration=2.331849659 podStartE2EDuration="2.804566674s" totalImagesPullingTime="472.717015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:52.802346669 +0000 UTC m=+1003.866290627" watchObservedRunningTime="2026-09-17 18:19:52.804566674 +0000 UTC m=+1003.868510632" Sep 17 18:19:53.403614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:53.403575 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:53.403831 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:53.403633 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:53.416531 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:53.416508 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:53.801755 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:53.801668 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:19:58.795874 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:19:58.795840 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:20:20.502787 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.502748 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz"] Sep 17 18:20:20.508263 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.508244 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.511679 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.511657 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-99ac10\"/\"openshift-service-ca.crt\"" Sep 17 18:20:20.511807 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.511657 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-99ac10\"/\"test-model-99ac10-kserve-self-signed-certs\"" Sep 17 18:20:20.511807 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.511713 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-99ac10\"/\"kube-root-ca.crt\"" Sep 17 18:20:20.511807 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.511685 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-99ac10\"/\"default-dockercfg-n2mrx\"" Sep 17 18:20:20.516488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.516466 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz"] Sep 17 18:20:20.572960 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.572926 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-model-cache\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.573137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.572982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-home\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.573137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.573033 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk7f2\" (UniqueName: \"kubernetes.io/projected/be463b6f-4968-4ad7-bef2-f68f51c82025-kube-api-access-nk7f2\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.573137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.573107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-tmp-dir\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.573327 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.573151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-dshm\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.573327 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.573289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be463b6f-4968-4ad7-bef2-f68f51c82025-tls-certs\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674204 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be463b6f-4968-4ad7-bef2-f68f51c82025-tls-certs\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674213 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-model-cache\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674239 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-home\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674275 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk7f2\" (UniqueName: \"kubernetes.io/projected/be463b6f-4968-4ad7-bef2-f68f51c82025-kube-api-access-nk7f2\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-tmp-dir\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-dshm\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674637 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-home\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674648 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-model-cache\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.674774 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.674713 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-tmp-dir\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.676482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.676458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-dshm\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.676734 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.676718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be463b6f-4968-4ad7-bef2-f68f51c82025-tls-certs\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.683205 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.683186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk7f2\" (UniqueName: \"kubernetes.io/projected/be463b6f-4968-4ad7-bef2-f68f51c82025-kube-api-access-nk7f2\") pod \"test-model-99ac10-kserve-5b7ffbbcf-5kmgz\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:20.819680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:20.819580 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:21.100432 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:21.100404 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz"] Sep 17 18:20:21.102163 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:20:21.102141 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe463b6f_4968_4ad7_bef2_f68f51c82025.slice/crio-d89205e86b0312b34fe1e5bfe1e3ea0c1fbd87436d9e7c2774fe53d1420decf6 WatchSource:0}: Error finding container d89205e86b0312b34fe1e5bfe1e3ea0c1fbd87436d9e7c2774fe53d1420decf6: Status 404 returned error can't find the container with id d89205e86b0312b34fe1e5bfe1e3ea0c1fbd87436d9e7c2774fe53d1420decf6 Sep 17 18:20:21.957115 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:21.957081 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" event={"ID":"be463b6f-4968-4ad7-bef2-f68f51c82025","Type":"ContainerStarted","Data":"68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da"} Sep 17 18:20:21.957531 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:21.957129 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" event={"ID":"be463b6f-4968-4ad7-bef2-f68f51c82025","Type":"ContainerStarted","Data":"d89205e86b0312b34fe1e5bfe1e3ea0c1fbd87436d9e7c2774fe53d1420decf6"} Sep 17 18:20:21.977470 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:21.977423 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" podStartSLOduration=1.9773865750000001 podStartE2EDuration="1.977386575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:21.977199034 +0000 UTC m=+1033.041142993" watchObservedRunningTime="2026-09-17 18:20:21.977386575 +0000 UTC m=+1033.041330532" Sep 17 18:20:30.820250 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:30.820209 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:30.820250 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:30.820260 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:30.833433 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:30.833402 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:31.021029 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:31.020995 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:20:52.318102 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.318065 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt"] Sep 17 18:20:52.322472 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.322448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.324929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.324906 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d62f3adc-serving-cert\"" Sep 17 18:20:52.332183 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.331882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt"] Sep 17 18:20:52.498854 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.498817 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-maas-api-tls\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.499016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.498918 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ada0c1-514a-44eb-9f11-65a149a1824e-openshift-service-ca\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.499016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.498949 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-metrics-certs\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.499016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.498966 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sgg4\" (UniqueName: \"kubernetes.io/projected/28ada0c1-514a-44eb-9f11-65a149a1824e-kube-api-access-8sgg4\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.599960 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.599853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-maas-api-tls\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.599960 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.599957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ada0c1-514a-44eb-9f11-65a149a1824e-openshift-service-ca\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.600194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.599990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-metrics-certs\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.600194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.600008 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sgg4\" (UniqueName: \"kubernetes.io/projected/28ada0c1-514a-44eb-9f11-65a149a1824e-kube-api-access-8sgg4\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.600659 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.600632 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ada0c1-514a-44eb-9f11-65a149a1824e-openshift-service-ca\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.602286 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.602263 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-maas-api-tls\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.602438 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.602418 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-metrics-certs\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.608529 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.608503 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sgg4\" (UniqueName: \"kubernetes.io/projected/28ada0c1-514a-44eb-9f11-65a149a1824e-kube-api-access-8sgg4\") pod \"maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.637108 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.636826 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:52.917262 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:52.917233 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt"] Sep 17 18:20:52.919054 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:20:52.919012 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ada0c1_514a_44eb_9f11_65a149a1824e.slice/crio-948cb957b538b7e7a59b5461e78640ddc318f2cad288ce9df4d341e1ada6849c WatchSource:0}: Error finding container 948cb957b538b7e7a59b5461e78640ddc318f2cad288ce9df4d341e1ada6849c: Status 404 returned error can't find the container with id 948cb957b538b7e7a59b5461e78640ddc318f2cad288ce9df4d341e1ada6849c Sep 17 18:20:53.150290 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:53.150258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" event={"ID":"28ada0c1-514a-44eb-9f11-65a149a1824e","Type":"ContainerStarted","Data":"948cb957b538b7e7a59b5461e78640ddc318f2cad288ce9df4d341e1ada6849c"} Sep 17 18:20:54.158592 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:54.158546 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" event={"ID":"28ada0c1-514a-44eb-9f11-65a149a1824e","Type":"ContainerStarted","Data":"c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54"} Sep 17 18:20:54.159005 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:54.158640 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:20:54.177543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:20:54.177495 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" podStartSLOduration=1.816105155 podStartE2EDuration="2.177482156s" totalImagesPullingTime="361.377001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:54.176285388 +0000 UTC m=+1065.240229347" watchObservedRunningTime="2026-09-17 18:20:54.177482156 +0000 UTC m=+1065.241426113" Sep 17 18:21:00.171444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:00.171416 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:21:23.356738 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.356703 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d"] Sep 17 18:21:23.357367 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.356954 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" podUID="1db9b14b-80cf-4fbf-9054-4092c8b87e6c" containerName="maas-api" containerID="cri-o://5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f" gracePeriod=30 Sep 17 18:21:23.616842 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.616781 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:21:23.720379 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.720348 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p958s\" (UniqueName: \"kubernetes.io/projected/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-kube-api-access-p958s\") pod \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " Sep 17 18:21:23.720583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.720389 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-metrics-certs\") pod \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " Sep 17 18:21:23.720583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.720452 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls\") pod \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " Sep 17 18:21:23.720713 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.720573 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-openshift-service-ca\") pod \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\" (UID: \"1db9b14b-80cf-4fbf-9054-4092c8b87e6c\") " Sep 17 18:21:23.720910 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.720881 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1db9b14b-80cf-4fbf-9054-4092c8b87e6c" (UID: "1db9b14b-80cf-4fbf-9054-4092c8b87e6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:21:23.721179 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.721151 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:21:23.722677 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.722650 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1db9b14b-80cf-4fbf-9054-4092c8b87e6c" (UID: "1db9b14b-80cf-4fbf-9054-4092c8b87e6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:21:23.722782 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.722751 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1db9b14b-80cf-4fbf-9054-4092c8b87e6c" (UID: "1db9b14b-80cf-4fbf-9054-4092c8b87e6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:21:23.723042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.723018 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-kube-api-access-p958s" (OuterVolumeSpecName: "kube-api-access-p958s") pod "1db9b14b-80cf-4fbf-9054-4092c8b87e6c" (UID: "1db9b14b-80cf-4fbf-9054-4092c8b87e6c"). InnerVolumeSpecName "kube-api-access-p958s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:23.821968 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.821936 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p958s\" (UniqueName: \"kubernetes.io/projected/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-kube-api-access-p958s\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:21:23.821968 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.821969 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:21:23.822176 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:23.821982 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1db9b14b-80cf-4fbf-9054-4092c8b87e6c-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:21:24.346043 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.346007 2582 generic.go:356] "Generic (PLEG): container finished" podID="1db9b14b-80cf-4fbf-9054-4092c8b87e6c" containerID="5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f" exitCode=0 Sep 17 18:21:24.346043 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.346051 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" Sep 17 18:21:24.346253 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.346100 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" event={"ID":"1db9b14b-80cf-4fbf-9054-4092c8b87e6c","Type":"ContainerDied","Data":"5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f"} Sep 17 18:21:24.346253 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.346159 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d" event={"ID":"1db9b14b-80cf-4fbf-9054-4092c8b87e6c","Type":"ContainerDied","Data":"66bdd0b72fab21929fab853c8f6765fd73b5ff866149ef014b4f7d89080aeb7b"} Sep 17 18:21:24.346253 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.346183 2582 scope.go:117] "RemoveContainer" containerID="5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f" Sep 17 18:21:24.356798 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.356779 2582 scope.go:117] "RemoveContainer" containerID="5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f" Sep 17 18:21:24.357114 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:21:24.357080 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f\": container with ID starting with 5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f not found: ID does not exist" containerID="5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f" Sep 17 18:21:24.357156 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.357110 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f"} err="failed to get container status \"5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f\": rpc error: code = NotFound desc = could not find container \"5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f\": container with ID starting with 5c3784dd418d920687077f2135bd978aac811048820ec29e53f5a2e933238d8f not found: ID does not exist" Sep 17 18:21:24.370807 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.370782 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d"] Sep 17 18:21:24.374900 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:24.374880 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-83507f-5c57547c49-phv5d"] Sep 17 18:21:25.348284 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:25.348252 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1db9b14b-80cf-4fbf-9054-4092c8b87e6c" path="/var/lib/kubelet/pods/1db9b14b-80cf-4fbf-9054-4092c8b87e6c/volumes" Sep 17 18:21:37.082392 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.082354 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt"] Sep 17 18:21:37.082988 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.082624 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" podUID="28ada0c1-514a-44eb-9f11-65a149a1824e" containerName="maas-api" containerID="cri-o://c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54" gracePeriod=30 Sep 17 18:21:37.359533 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.359501 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:21:37.363947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.363923 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sgg4\" (UniqueName: \"kubernetes.io/projected/28ada0c1-514a-44eb-9f11-65a149a1824e-kube-api-access-8sgg4\") pod \"28ada0c1-514a-44eb-9f11-65a149a1824e\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " Sep 17 18:21:37.364058 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.363969 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ada0c1-514a-44eb-9f11-65a149a1824e-openshift-service-ca\") pod \"28ada0c1-514a-44eb-9f11-65a149a1824e\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " Sep 17 18:21:37.364058 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.364052 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-metrics-certs\") pod \"28ada0c1-514a-44eb-9f11-65a149a1824e\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " Sep 17 18:21:37.364132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.364105 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-maas-api-tls\") pod \"28ada0c1-514a-44eb-9f11-65a149a1824e\" (UID: \"28ada0c1-514a-44eb-9f11-65a149a1824e\") " Sep 17 18:21:37.364388 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.364363 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ada0c1-514a-44eb-9f11-65a149a1824e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28ada0c1-514a-44eb-9f11-65a149a1824e" (UID: "28ada0c1-514a-44eb-9f11-65a149a1824e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:21:37.364524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.364449 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28ada0c1-514a-44eb-9f11-65a149a1824e-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:21:37.366145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.366120 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "28ada0c1-514a-44eb-9f11-65a149a1824e" (UID: "28ada0c1-514a-44eb-9f11-65a149a1824e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:21:37.366279 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.366255 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28ada0c1-514a-44eb-9f11-65a149a1824e" (UID: "28ada0c1-514a-44eb-9f11-65a149a1824e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:21:37.366352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.366273 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ada0c1-514a-44eb-9f11-65a149a1824e-kube-api-access-8sgg4" (OuterVolumeSpecName: "kube-api-access-8sgg4") pod "28ada0c1-514a-44eb-9f11-65a149a1824e" (UID: "28ada0c1-514a-44eb-9f11-65a149a1824e"). InnerVolumeSpecName "kube-api-access-8sgg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:37.427694 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.427661 2582 generic.go:356] "Generic (PLEG): container finished" podID="28ada0c1-514a-44eb-9f11-65a149a1824e" containerID="c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54" exitCode=0 Sep 17 18:21:37.427896 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.427722 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" Sep 17 18:21:37.427896 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.427746 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" event={"ID":"28ada0c1-514a-44eb-9f11-65a149a1824e","Type":"ContainerDied","Data":"c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54"} Sep 17 18:21:37.427896 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.427798 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt" event={"ID":"28ada0c1-514a-44eb-9f11-65a149a1824e","Type":"ContainerDied","Data":"948cb957b538b7e7a59b5461e78640ddc318f2cad288ce9df4d341e1ada6849c"} Sep 17 18:21:37.427896 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.427820 2582 scope.go:117] "RemoveContainer" containerID="c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54" Sep 17 18:21:37.446105 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.446084 2582 scope.go:117] "RemoveContainer" containerID="c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54" Sep 17 18:21:37.446659 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:21:37.446629 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54\": container with ID starting with c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54 not found: ID does not exist" containerID="c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54" Sep 17 18:21:37.446659 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.446667 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54"} err="failed to get container status \"c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54\": rpc error: code = NotFound desc = could not find container \"c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54\": container with ID starting with c4af0f97b2af0472cb370241742098bda31a0c228fcd3690268cdd8d772d4d54 not found: ID does not exist" Sep 17 18:21:37.461455 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.461420 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt"] Sep 17 18:21:37.463839 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.463812 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d62f3adc-6cc74fcd86-4jgpt"] Sep 17 18:21:37.465102 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.465076 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:21:37.465102 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.465099 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28ada0c1-514a-44eb-9f11-65a149a1824e-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:21:37.465225 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:37.465112 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sgg4\" (UniqueName: \"kubernetes.io/projected/28ada0c1-514a-44eb-9f11-65a149a1824e-kube-api-access-8sgg4\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:21:39.348407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:21:39.348366 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ada0c1-514a-44eb-9f11-65a149a1824e" path="/var/lib/kubelet/pods/28ada0c1-514a-44eb-9f11-65a149a1824e/volumes" Sep 17 18:22:05.415766 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.415726 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx"] Sep 17 18:22:05.416491 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.416469 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ada0c1-514a-44eb-9f11-65a149a1824e" containerName="maas-api" Sep 17 18:22:05.416491 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.416493 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ada0c1-514a-44eb-9f11-65a149a1824e" containerName="maas-api" Sep 17 18:22:05.416657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.416510 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1db9b14b-80cf-4fbf-9054-4092c8b87e6c" containerName="maas-api" Sep 17 18:22:05.416657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.416518 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1db9b14b-80cf-4fbf-9054-4092c8b87e6c" containerName="maas-api" Sep 17 18:22:05.416657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.416644 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ada0c1-514a-44eb-9f11-65a149a1824e" containerName="maas-api" Sep 17 18:22:05.416810 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.416660 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1db9b14b-80cf-4fbf-9054-4092c8b87e6c" containerName="maas-api" Sep 17 18:22:05.420193 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.420169 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.422432 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.422261 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-1a9245-serving-cert\"" Sep 17 18:22:05.430919 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.430433 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx"] Sep 17 18:22:05.546361 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.546322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.546538 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.546389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c540d39a-9710-4b13-b3f8-2d53cf955701-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.546538 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.546440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwqn\" (UniqueName: \"kubernetes.io/projected/c540d39a-9710-4b13-b3f8-2d53cf955701-kube-api-access-xqwqn\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.546538 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.546524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.647375 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.647337 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c540d39a-9710-4b13-b3f8-2d53cf955701-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.647575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.647398 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwqn\" (UniqueName: \"kubernetes.io/projected/c540d39a-9710-4b13-b3f8-2d53cf955701-kube-api-access-xqwqn\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.647575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.647497 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.647575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.647562 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.648236 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.648205 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c540d39a-9710-4b13-b3f8-2d53cf955701-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.650169 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.650146 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.650169 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.650155 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.655170 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.655149 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwqn\" (UniqueName: \"kubernetes.io/projected/c540d39a-9710-4b13-b3f8-2d53cf955701-kube-api-access-xqwqn\") pod \"maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:05.735428 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:05.735330 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:06.018230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:06.018154 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx"] Sep 17 18:22:06.021721 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:22:06.021692 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc540d39a_9710_4b13_b3f8_2d53cf955701.slice/crio-f32c6d0ab2b4813a9b2adc4f252dc62ce99a3ca572dc2b59f678d5b3faff9309 WatchSource:0}: Error finding container f32c6d0ab2b4813a9b2adc4f252dc62ce99a3ca572dc2b59f678d5b3faff9309: Status 404 returned error can't find the container with id f32c6d0ab2b4813a9b2adc4f252dc62ce99a3ca572dc2b59f678d5b3faff9309 Sep 17 18:22:06.612367 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:06.612327 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" event={"ID":"c540d39a-9710-4b13-b3f8-2d53cf955701","Type":"ContainerStarted","Data":"bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166"} Sep 17 18:22:06.612743 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:06.612421 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" event={"ID":"c540d39a-9710-4b13-b3f8-2d53cf955701","Type":"ContainerStarted","Data":"f32c6d0ab2b4813a9b2adc4f252dc62ce99a3ca572dc2b59f678d5b3faff9309"} Sep 17 18:22:06.612743 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:06.612449 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:06.633204 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:06.633153 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" podStartSLOduration=1.135765478 podStartE2EDuration="1.633136132s" totalImagesPullingTime="497.370654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:06.630190298 +0000 UTC m=+1137.694134256" watchObservedRunningTime="2026-09-17 18:22:06.633136132 +0000 UTC m=+1137.697080090" Sep 17 18:22:11.186261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.186221 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn"] Sep 17 18:22:11.190472 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.190448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.192833 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.192807 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8c963b\"/\"openshift-service-ca.crt\"" Sep 17 18:22:11.193820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.193800 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8c963b\"/\"kube-root-ca.crt\"" Sep 17 18:22:11.193952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.193890 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8c963b\"/\"default-dockercfg-n97nb\"" Sep 17 18:22:11.193952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.193898 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8c963b\"/\"noten-model-8c963b-kserve-self-signed-certs\"" Sep 17 18:22:11.197843 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.197822 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn"] Sep 17 18:22:11.206613 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.206591 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whx42\" (UniqueName: \"kubernetes.io/projected/b03728de-721d-4510-aa53-2d776e780541-kube-api-access-whx42\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.206703 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.206634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-model-cache\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.206703 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.206665 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-home\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.206703 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.206689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-tmp-dir\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.206811 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.206769 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-dshm\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.206878 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.206831 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03728de-721d-4510-aa53-2d776e780541-tls-certs\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.307948 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.307912 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-dshm\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.308129 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.307955 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03728de-721d-4510-aa53-2d776e780541-tls-certs\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.308129 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.308059 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whx42\" (UniqueName: \"kubernetes.io/projected/b03728de-721d-4510-aa53-2d776e780541-kube-api-access-whx42\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.308129 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.308108 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-model-cache\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.308309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.308142 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-home\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.308309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.308163 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-tmp-dir\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.308478 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.308458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-model-cache\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.308550 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.308497 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-home\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.308550 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.308540 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-tmp-dir\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.310180 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.310154 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-dshm\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.310448 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.310429 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03728de-721d-4510-aa53-2d776e780541-tls-certs\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.316509 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.316490 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whx42\" (UniqueName: \"kubernetes.io/projected/b03728de-721d-4510-aa53-2d776e780541-kube-api-access-whx42\") pod \"noten-model-8c963b-kserve-7d5655c699-ccpnn\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.501283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.501205 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:11.783988 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:11.783959 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn"] Sep 17 18:22:11.785758 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:22:11.785727 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb03728de_721d_4510_aa53_2d776e780541.slice/crio-5119445a4cea96592b106323faebcc8267edaf74133886c05167ab66b427edeb WatchSource:0}: Error finding container 5119445a4cea96592b106323faebcc8267edaf74133886c05167ab66b427edeb: Status 404 returned error can't find the container with id 5119445a4cea96592b106323faebcc8267edaf74133886c05167ab66b427edeb Sep 17 18:22:12.624356 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:12.624325 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:22:12.651898 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:12.651830 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" event={"ID":"b03728de-721d-4510-aa53-2d776e780541","Type":"ContainerStarted","Data":"b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb"} Sep 17 18:22:12.652093 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:12.651905 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" event={"ID":"b03728de-721d-4510-aa53-2d776e780541","Type":"ContainerStarted","Data":"5119445a4cea96592b106323faebcc8267edaf74133886c05167ab66b427edeb"} Sep 17 18:22:12.670168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:12.670113 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" podStartSLOduration=1.670096655 podStartE2EDuration="1.670096655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:12.668593848 +0000 UTC m=+1143.732537807" watchObservedRunningTime="2026-09-17 18:22:12.670096655 +0000 UTC m=+1143.734040612" Sep 17 18:22:21.502354 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:21.502319 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:21.502732 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:21.502376 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:21.515010 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:21.514983 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:21.715787 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:21.715759 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:22:40.008814 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.008730 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv"] Sep 17 18:22:40.012622 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.012604 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.014969 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.014936 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-03270e-serving-cert\"" Sep 17 18:22:40.022281 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.022118 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv"] Sep 17 18:22:40.119221 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.119182 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.119393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.119226 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.119393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.119349 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvc89\" (UniqueName: \"kubernetes.io/projected/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-kube-api-access-lvc89\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.119501 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.119396 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.220920 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.220887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.221098 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.221002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvc89\" (UniqueName: \"kubernetes.io/projected/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-kube-api-access-lvc89\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.221098 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.221050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.221180 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.221155 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.221596 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.221569 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.223581 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.223552 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.223676 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.223583 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.228590 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.228568 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvc89\" (UniqueName: \"kubernetes.io/projected/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-kube-api-access-lvc89\") pod \"maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.327090 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.327014 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:40.605372 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.605343 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv"] Sep 17 18:22:40.607109 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:22:40.607076 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd868cde7_cbc3_4a46_b7e0_f45d2dcd5bec.slice/crio-e09fe6150496d3ccd90c1f0ecbc48cfb4243bd8d263b49ebdcada7398eb60fb5 WatchSource:0}: Error finding container e09fe6150496d3ccd90c1f0ecbc48cfb4243bd8d263b49ebdcada7398eb60fb5: Status 404 returned error can't find the container with id e09fe6150496d3ccd90c1f0ecbc48cfb4243bd8d263b49ebdcada7398eb60fb5 Sep 17 18:22:40.830214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:40.830178 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" event={"ID":"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec","Type":"ContainerStarted","Data":"e09fe6150496d3ccd90c1f0ecbc48cfb4243bd8d263b49ebdcada7398eb60fb5"} Sep 17 18:22:41.838235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:41.838194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" event={"ID":"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec","Type":"ContainerStarted","Data":"573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f"} Sep 17 18:22:41.838681 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:41.838358 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:41.858161 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:41.858107 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" podStartSLOduration=2.530728553 podStartE2EDuration="2.858092167s" totalImagesPullingTime="327.363614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:41.855746547 +0000 UTC m=+1172.919690518" watchObservedRunningTime="2026-09-17 18:22:41.858092167 +0000 UTC m=+1172.922036125" Sep 17 18:22:47.850555 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:47.850525 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:22:49.577011 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:22:49.576976 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn"] Sep 17 18:23:04.598110 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.598053 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" podUID="b03728de-721d-4510-aa53-2d776e780541" containerName="main" containerID="cri-o://b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb" gracePeriod=45 Sep 17 18:23:04.869086 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.869062 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:23:04.977888 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.977845 2582 generic.go:356] "Generic (PLEG): container finished" podID="b03728de-721d-4510-aa53-2d776e780541" containerID="b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb" exitCode=0 Sep 17 18:23:04.978051 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.977912 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" Sep 17 18:23:04.978051 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.977933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" event={"ID":"b03728de-721d-4510-aa53-2d776e780541","Type":"ContainerDied","Data":"b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb"} Sep 17 18:23:04.978051 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.977981 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn" event={"ID":"b03728de-721d-4510-aa53-2d776e780541","Type":"ContainerDied","Data":"5119445a4cea96592b106323faebcc8267edaf74133886c05167ab66b427edeb"} Sep 17 18:23:04.978051 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.977998 2582 scope.go:117] "RemoveContainer" containerID="b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb" Sep 17 18:23:04.980983 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.980956 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whx42\" (UniqueName: \"kubernetes.io/projected/b03728de-721d-4510-aa53-2d776e780541-kube-api-access-whx42\") pod \"b03728de-721d-4510-aa53-2d776e780541\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " Sep 17 18:23:04.981128 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981009 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-home\") pod \"b03728de-721d-4510-aa53-2d776e780541\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " Sep 17 18:23:04.981128 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981071 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-dshm\") pod \"b03728de-721d-4510-aa53-2d776e780541\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " Sep 17 18:23:04.981128 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981118 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-model-cache\") pod \"b03728de-721d-4510-aa53-2d776e780541\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " Sep 17 18:23:04.981308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981193 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03728de-721d-4510-aa53-2d776e780541-tls-certs\") pod \"b03728de-721d-4510-aa53-2d776e780541\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " Sep 17 18:23:04.981308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981257 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-tmp-dir\") pod \"b03728de-721d-4510-aa53-2d776e780541\" (UID: \"b03728de-721d-4510-aa53-2d776e780541\") " Sep 17 18:23:04.981594 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981560 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b03728de-721d-4510-aa53-2d776e780541" (UID: "b03728de-721d-4510-aa53-2d776e780541"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:04.981708 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981570 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-home" (OuterVolumeSpecName: "home") pod "b03728de-721d-4510-aa53-2d776e780541" (UID: "b03728de-721d-4510-aa53-2d776e780541"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:04.981708 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981653 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-model-cache" (OuterVolumeSpecName: "model-cache") pod "b03728de-721d-4510-aa53-2d776e780541" (UID: "b03728de-721d-4510-aa53-2d776e780541"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:04.981840 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981704 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:04.981840 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.981725 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:04.983556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.983522 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-dshm" (OuterVolumeSpecName: "dshm") pod "b03728de-721d-4510-aa53-2d776e780541" (UID: "b03728de-721d-4510-aa53-2d776e780541"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:04.983556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.983537 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03728de-721d-4510-aa53-2d776e780541-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b03728de-721d-4510-aa53-2d776e780541" (UID: "b03728de-721d-4510-aa53-2d776e780541"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:23:04.983700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.983614 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03728de-721d-4510-aa53-2d776e780541-kube-api-access-whx42" (OuterVolumeSpecName: "kube-api-access-whx42") pod "b03728de-721d-4510-aa53-2d776e780541" (UID: "b03728de-721d-4510-aa53-2d776e780541"). InnerVolumeSpecName "kube-api-access-whx42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:04.989653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.988817 2582 scope.go:117] "RemoveContainer" containerID="b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb" Sep 17 18:23:04.991278 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:23:04.990679 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb\": container with ID starting with b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb not found: ID does not exist" containerID="b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb" Sep 17 18:23:04.991278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:04.991243 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb"} err="failed to get container status \"b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb\": rpc error: code = NotFound desc = could not find container \"b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb\": container with ID starting with b42d89cf1c124f3d45d4c6e6256dd37d933bc5084f96d737e4ade4deb22867fb not found: ID does not exist" Sep 17 18:23:05.083259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:05.083226 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03728de-721d-4510-aa53-2d776e780541-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:05.083259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:05.083255 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whx42\" (UniqueName: \"kubernetes.io/projected/b03728de-721d-4510-aa53-2d776e780541-kube-api-access-whx42\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:05.083259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:05.083265 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:05.083470 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:05.083275 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03728de-721d-4510-aa53-2d776e780541-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:05.302095 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:05.302065 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn"] Sep 17 18:23:05.304488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:05.304463 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-8c963b/noten-model-8c963b-kserve-7d5655c699-ccpnn"] Sep 17 18:23:05.349039 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:05.349012 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03728de-721d-4510-aa53-2d776e780541" path="/var/lib/kubelet/pods/b03728de-721d-4510-aa53-2d776e780541/volumes" Sep 17 18:23:11.328639 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:11.328612 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:23:11.331856 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:11.331831 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:23:11.346662 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:11.346643 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:23:11.350045 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:11.350025 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:23:12.559901 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.559677 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5"] Sep 17 18:23:12.560376 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.560253 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" podUID="8633d73b-d1c9-431a-ae39-b3aa4b81a431" containerName="maas-api" containerID="cri-o://a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c" gracePeriod=30 Sep 17 18:23:12.614888 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.614820 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml"] Sep 17 18:23:12.615164 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.615132 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" podUID="017fb4d2-88d2-4585-a390-3cd31221516a" containerName="payload-processing" containerID="cri-o://79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04" gracePeriod=30 Sep 17 18:23:12.617036 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.617010 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4"] Sep 17 18:23:12.914229 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.914196 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:23:12.933292 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.933046 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:23:12.968610 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.968572 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633d73b-d1c9-431a-ae39-b3aa4b81a431-openshift-service-ca\") pod \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " Sep 17 18:23:12.968819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.968659 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-maas-api-tls\") pod \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " Sep 17 18:23:12.968819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.968687 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017fb4d2-88d2-4585-a390-3cd31221516a-plugins-config-volume\") pod \"017fb4d2-88d2-4585-a390-3cd31221516a\" (UID: \"017fb4d2-88d2-4585-a390-3cd31221516a\") " Sep 17 18:23:12.968819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.968722 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q92pw\" (UniqueName: \"kubernetes.io/projected/017fb4d2-88d2-4585-a390-3cd31221516a-kube-api-access-q92pw\") pod \"017fb4d2-88d2-4585-a390-3cd31221516a\" (UID: \"017fb4d2-88d2-4585-a390-3cd31221516a\") " Sep 17 18:23:12.969021 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.968819 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-metrics-certs\") pod \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " Sep 17 18:23:12.969021 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.968854 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqhs9\" (UniqueName: \"kubernetes.io/projected/8633d73b-d1c9-431a-ae39-b3aa4b81a431-kube-api-access-mqhs9\") pod \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\" (UID: \"8633d73b-d1c9-431a-ae39-b3aa4b81a431\") " Sep 17 18:23:12.970069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.970038 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017fb4d2-88d2-4585-a390-3cd31221516a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "017fb4d2-88d2-4585-a390-3cd31221516a" (UID: "017fb4d2-88d2-4585-a390-3cd31221516a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:23:12.970473 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.970447 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8633d73b-d1c9-431a-ae39-b3aa4b81a431-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8633d73b-d1c9-431a-ae39-b3aa4b81a431" (UID: "8633d73b-d1c9-431a-ae39-b3aa4b81a431"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:23:12.971938 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.971909 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8633d73b-d1c9-431a-ae39-b3aa4b81a431-kube-api-access-mqhs9" (OuterVolumeSpecName: "kube-api-access-mqhs9") pod "8633d73b-d1c9-431a-ae39-b3aa4b81a431" (UID: "8633d73b-d1c9-431a-ae39-b3aa4b81a431"). InnerVolumeSpecName "kube-api-access-mqhs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:12.973003 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.972974 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8633d73b-d1c9-431a-ae39-b3aa4b81a431" (UID: "8633d73b-d1c9-431a-ae39-b3aa4b81a431"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:23:12.973289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.973270 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8633d73b-d1c9-431a-ae39-b3aa4b81a431" (UID: "8633d73b-d1c9-431a-ae39-b3aa4b81a431"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:23:12.973609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:12.973592 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017fb4d2-88d2-4585-a390-3cd31221516a-kube-api-access-q92pw" (OuterVolumeSpecName: "kube-api-access-q92pw") pod "017fb4d2-88d2-4585-a390-3cd31221516a" (UID: "017fb4d2-88d2-4585-a390-3cd31221516a"). InnerVolumeSpecName "kube-api-access-q92pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:13.027960 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.027928 2582 generic.go:356] "Generic (PLEG): container finished" podID="8633d73b-d1c9-431a-ae39-b3aa4b81a431" containerID="a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c" exitCode=0 Sep 17 18:23:13.028119 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.027996 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" Sep 17 18:23:13.028119 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.028023 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" event={"ID":"8633d73b-d1c9-431a-ae39-b3aa4b81a431","Type":"ContainerDied","Data":"a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c"} Sep 17 18:23:13.028119 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.028117 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5" event={"ID":"8633d73b-d1c9-431a-ae39-b3aa4b81a431","Type":"ContainerDied","Data":"95b5072e32b42388400ecacfae8523c10c43ec60fce63b7f03fd09c9cc997941"} Sep 17 18:23:13.028292 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.028149 2582 scope.go:117] "RemoveContainer" containerID="a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c" Sep 17 18:23:13.029647 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.029623 2582 generic.go:356] "Generic (PLEG): container finished" podID="017fb4d2-88d2-4585-a390-3cd31221516a" containerID="79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04" exitCode=0 Sep 17 18:23:13.029781 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.029667 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" Sep 17 18:23:13.029781 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.029736 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" event={"ID":"017fb4d2-88d2-4585-a390-3cd31221516a","Type":"ContainerDied","Data":"79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04"} Sep 17 18:23:13.029781 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.029773 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml" event={"ID":"017fb4d2-88d2-4585-a390-3cd31221516a","Type":"ContainerDied","Data":"1fac063c429a33499a0a33b7197694bf32e3d33044779bd839e9f17faf4878ae"} Sep 17 18:23:13.030027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.030001 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" podUID="6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71" containerName="payload-pre-processing" containerID="cri-o://0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd" gracePeriod=30 Sep 17 18:23:13.047136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.040005 2582 scope.go:117] "RemoveContainer" containerID="a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c" Sep 17 18:23:13.047136 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:23:13.040328 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c\": container with ID starting with a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c not found: ID does not exist" containerID="a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c" Sep 17 18:23:13.047136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.040361 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c"} err="failed to get container status \"a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c\": rpc error: code = NotFound desc = could not find container \"a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c\": container with ID starting with a9fb0e5a2ef06bf12bc98c47cb0c9efdec56d04c66a335c10b1fb6ce9ea7873c not found: ID does not exist" Sep 17 18:23:13.047136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.040386 2582 scope.go:117] "RemoveContainer" containerID="79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04" Sep 17 18:23:13.069968 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.069941 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:13.069968 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.069964 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqhs9\" (UniqueName: \"kubernetes.io/projected/8633d73b-d1c9-431a-ae39-b3aa4b81a431-kube-api-access-mqhs9\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:13.069968 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.069974 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8633d73b-d1c9-431a-ae39-b3aa4b81a431-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:13.070247 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.069983 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8633d73b-d1c9-431a-ae39-b3aa4b81a431-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:13.070247 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.069992 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/017fb4d2-88d2-4585-a390-3cd31221516a-plugins-config-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:13.070247 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.070000 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q92pw\" (UniqueName: \"kubernetes.io/projected/017fb4d2-88d2-4585-a390-3cd31221516a-kube-api-access-q92pw\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:13.098384 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.098366 2582 scope.go:117] "RemoveContainer" containerID="79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04" Sep 17 18:23:13.098717 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:23:13.098695 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04\": container with ID starting with 79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04 not found: ID does not exist" containerID="79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04" Sep 17 18:23:13.098785 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.098724 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04"} err="failed to get container status \"79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04\": rpc error: code = NotFound desc = could not find container \"79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04\": container with ID starting with 79e3903a657ff27e8457a75850b53125ad6b2325410f39393cbf168af6ac3e04 not found: ID does not exist" Sep 17 18:23:13.106980 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.106954 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5"] Sep 17 18:23:13.110542 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.110521 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c31adf-68b59d9b5f-dvlx5"] Sep 17 18:23:13.123131 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.123110 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml"] Sep 17 18:23:13.126444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.126418 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c31adf-5455b445cc-5mmml"] Sep 17 18:23:13.294235 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.294210 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:23:13.348486 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.348385 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017fb4d2-88d2-4585-a390-3cd31221516a" path="/var/lib/kubelet/pods/017fb4d2-88d2-4585-a390-3cd31221516a/volumes" Sep 17 18:23:13.348981 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.348958 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8633d73b-d1c9-431a-ae39-b3aa4b81a431" path="/var/lib/kubelet/pods/8633d73b-d1c9-431a-ae39-b3aa4b81a431/volumes" Sep 17 18:23:13.373245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.373223 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzpbr\" (UniqueName: \"kubernetes.io/projected/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-kube-api-access-xzpbr\") pod \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\" (UID: \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\") " Sep 17 18:23:13.373378 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.373353 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-plugins-config-volume\") pod \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\" (UID: \"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71\") " Sep 17 18:23:13.374079 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.373748 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71" (UID: "6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:23:13.374079 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.373984 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-plugins-config-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:13.375779 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.375760 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-kube-api-access-xzpbr" (OuterVolumeSpecName: "kube-api-access-xzpbr") pod "6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71" (UID: "6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71"). InnerVolumeSpecName "kube-api-access-xzpbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:13.474593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:13.474566 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzpbr\" (UniqueName: \"kubernetes.io/projected/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71-kube-api-access-xzpbr\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:14.039125 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.039089 2582 generic.go:356] "Generic (PLEG): container finished" podID="6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71" containerID="0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd" exitCode=0 Sep 17 18:23:14.039568 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.039145 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" Sep 17 18:23:14.039568 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.039172 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" event={"ID":"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71","Type":"ContainerDied","Data":"0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd"} Sep 17 18:23:14.039568 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.039246 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4" event={"ID":"6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71","Type":"ContainerDied","Data":"be5c9da9c8a1cc28be54df8149ffad4070c91e1b0ad6cf0ba17c3c38a777adc5"} Sep 17 18:23:14.039568 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.039286 2582 scope.go:117] "RemoveContainer" containerID="0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd" Sep 17 18:23:14.062964 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.062928 2582 scope.go:117] "RemoveContainer" containerID="0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd" Sep 17 18:23:14.063349 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:23:14.063319 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd\": container with ID starting with 0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd not found: ID does not exist" containerID="0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd" Sep 17 18:23:14.063423 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.063369 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd"} err="failed to get container status \"0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd\": rpc error: code = NotFound desc = could not find container \"0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd\": container with ID starting with 0f0c2e5c582b4ca39d2e647e460c382e70f54a0a6693fb874555eb32435316dd not found: ID does not exist" Sep 17 18:23:14.068192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.068161 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4"] Sep 17 18:23:14.071215 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:14.071193 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c31adf-7d667f4f4-hdxk4"] Sep 17 18:23:15.347674 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:15.347643 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71" path="/var/lib/kubelet/pods/6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71/volumes" Sep 17 18:23:23.284974 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.284910 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294"] Sep 17 18:23:23.285662 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285640 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71" containerName="payload-pre-processing" Sep 17 18:23:23.285752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285665 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71" containerName="payload-pre-processing" Sep 17 18:23:23.285752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285675 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017fb4d2-88d2-4585-a390-3cd31221516a" containerName="payload-processing" Sep 17 18:23:23.285752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285683 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="017fb4d2-88d2-4585-a390-3cd31221516a" containerName="payload-processing" Sep 17 18:23:23.285752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285706 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03728de-721d-4510-aa53-2d776e780541" containerName="main" Sep 17 18:23:23.285752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285714 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03728de-721d-4510-aa53-2d776e780541" containerName="main" Sep 17 18:23:23.285752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285729 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8633d73b-d1c9-431a-ae39-b3aa4b81a431" containerName="maas-api" Sep 17 18:23:23.285752 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285738 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8633d73b-d1c9-431a-ae39-b3aa4b81a431" containerName="maas-api" Sep 17 18:23:23.286130 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285876 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e9c824f-ab2c-4a7c-ad45-7d2f8ac34c71" containerName="payload-pre-processing" Sep 17 18:23:23.286130 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285896 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03728de-721d-4510-aa53-2d776e780541" containerName="main" Sep 17 18:23:23.286130 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285907 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8633d73b-d1c9-431a-ae39-b3aa4b81a431" containerName="maas-api" Sep 17 18:23:23.286130 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.285920 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="017fb4d2-88d2-4585-a390-3cd31221516a" containerName="payload-processing" Sep 17 18:23:23.290674 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.290653 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.292989 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.292965 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"e2e-conflict-b-92400e60-kserve-self-signed-certs\"" Sep 17 18:23:23.301631 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.301607 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294"] Sep 17 18:23:23.383212 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.383180 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86461fe-234f-4bab-a7bd-2dceb03f504d-tls-certs\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.383368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.383220 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-tmp-dir\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.383368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.383245 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-home\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.383368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.383339 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-model-cache\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.383476 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.383413 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-dshm\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.383476 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.383467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dd4z\" (UniqueName: \"kubernetes.io/projected/d86461fe-234f-4bab-a7bd-2dceb03f504d-kube-api-access-8dd4z\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.485042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.484993 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dd4z\" (UniqueName: \"kubernetes.io/projected/d86461fe-234f-4bab-a7bd-2dceb03f504d-kube-api-access-8dd4z\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.485225 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.485082 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86461fe-234f-4bab-a7bd-2dceb03f504d-tls-certs\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.485225 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.485129 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-tmp-dir\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.485225 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.485165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-home\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.485225 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.485215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-model-cache\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.485454 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.485276 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-dshm\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.486612 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.485983 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-tmp-dir\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.486612 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.486549 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-home\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.487066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.486762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-model-cache\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.488081 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.488057 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-dshm\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.490359 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.490330 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86461fe-234f-4bab-a7bd-2dceb03f504d-tls-certs\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.490997 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.490976 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v"] Sep 17 18:23:23.495391 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.495249 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.497508 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.497490 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d245d137-serving-cert\"" Sep 17 18:23:23.498609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.498585 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dd4z\" (UniqueName: \"kubernetes.io/projected/d86461fe-234f-4bab-a7bd-2dceb03f504d-kube-api-access-8dd4z\") pod \"e2e-conflict-b-92400e60-kserve-84f99957fc-5c294\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.509696 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.509602 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v"] Sep 17 18:23:23.586415 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.586331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-maas-api-tls\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.586415 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.586389 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-metrics-certs\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.586618 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.586509 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xcl\" (UniqueName: \"kubernetes.io/projected/9be6fab0-c4ad-483f-8f6b-b127f8b04615-kube-api-access-79xcl\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.586618 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.586606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be6fab0-c4ad-483f-8f6b-b127f8b04615-openshift-service-ca\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.600108 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.600076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:23.687447 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.687404 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be6fab0-c4ad-483f-8f6b-b127f8b04615-openshift-service-ca\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.687622 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.687546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-maas-api-tls\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.687622 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.687594 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-metrics-certs\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.687750 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.687647 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xcl\" (UniqueName: \"kubernetes.io/projected/9be6fab0-c4ad-483f-8f6b-b127f8b04615-kube-api-access-79xcl\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.688299 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.688265 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be6fab0-c4ad-483f-8f6b-b127f8b04615-openshift-service-ca\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.690144 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.690119 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-maas-api-tls\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.690360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.690343 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-metrics-certs\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.696418 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.696399 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xcl\" (UniqueName: \"kubernetes.io/projected/9be6fab0-c4ad-483f-8f6b-b127f8b04615-kube-api-access-79xcl\") pod \"maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.827056 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.827025 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:23.875079 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:23.875049 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294"] Sep 17 18:23:23.876393 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:23:23.876365 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd86461fe_234f_4bab_a7bd_2dceb03f504d.slice/crio-afc6e6b1e6dfebe11874e91d2e79739bacd310bcaf18f557f4fd8dcb573fdbc7 WatchSource:0}: Error finding container afc6e6b1e6dfebe11874e91d2e79739bacd310bcaf18f557f4fd8dcb573fdbc7: Status 404 returned error can't find the container with id afc6e6b1e6dfebe11874e91d2e79739bacd310bcaf18f557f4fd8dcb573fdbc7 Sep 17 18:23:24.108911 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:24.108219 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v"] Sep 17 18:23:24.120318 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:24.120286 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" event={"ID":"9be6fab0-c4ad-483f-8f6b-b127f8b04615","Type":"ContainerStarted","Data":"d316a291ec3f0bcfeb1a29b686d906dd55b94b18f3f1d4d97ba5e877d9c0d791"} Sep 17 18:23:24.121895 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:24.121856 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" event={"ID":"d86461fe-234f-4bab-a7bd-2dceb03f504d","Type":"ContainerStarted","Data":"574d5d6bb42bc7f0d297ad1f9d4f41799786ad6100b987ea91592a2c05c8ab33"} Sep 17 18:23:24.121895 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:24.121906 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" event={"ID":"d86461fe-234f-4bab-a7bd-2dceb03f504d","Type":"ContainerStarted","Data":"afc6e6b1e6dfebe11874e91d2e79739bacd310bcaf18f557f4fd8dcb573fdbc7"} Sep 17 18:23:24.145958 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:24.145911 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" podStartSLOduration=1.145895372 podStartE2EDuration="1.145895372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:24.143281811 +0000 UTC m=+1215.207225767" watchObservedRunningTime="2026-09-17 18:23:24.145895372 +0000 UTC m=+1215.209839330" Sep 17 18:23:25.129642 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:25.129598 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" event={"ID":"9be6fab0-c4ad-483f-8f6b-b127f8b04615","Type":"ContainerStarted","Data":"d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3"} Sep 17 18:23:25.130042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:25.129775 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:25.148520 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:25.148464 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" podStartSLOduration=1.6653939690000001 podStartE2EDuration="2.148444212s" totalImagesPullingTime="483.050243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:25.145808133 +0000 UTC m=+1216.209752085" watchObservedRunningTime="2026-09-17 18:23:25.148444212 +0000 UTC m=+1216.212388172" Sep 17 18:23:28.780554 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:28.780356 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz"] Sep 17 18:23:31.099716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.099683 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd"] Sep 17 18:23:31.100256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.099926 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" podUID="e46a520c-3553-4a0f-a997-e22b2ba0eda1" containerName="maas-api" containerID="cri-o://97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7" gracePeriod=30 Sep 17 18:23:31.144107 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.144079 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:23:31.374524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.374498 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:23:31.473256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.473228 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-metrics-certs\") pod \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " Sep 17 18:23:31.473256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.473260 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46a520c-3553-4a0f-a997-e22b2ba0eda1-openshift-service-ca\") pod \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " Sep 17 18:23:31.473459 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.473333 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95n6k\" (UniqueName: \"kubernetes.io/projected/e46a520c-3553-4a0f-a997-e22b2ba0eda1-kube-api-access-95n6k\") pod \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " Sep 17 18:23:31.473516 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.473476 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls\") pod \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\" (UID: \"e46a520c-3553-4a0f-a997-e22b2ba0eda1\") " Sep 17 18:23:31.473667 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.473643 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46a520c-3553-4a0f-a997-e22b2ba0eda1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e46a520c-3553-4a0f-a997-e22b2ba0eda1" (UID: "e46a520c-3553-4a0f-a997-e22b2ba0eda1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:23:31.474250 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.474195 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e46a520c-3553-4a0f-a997-e22b2ba0eda1-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:31.476196 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.476140 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e46a520c-3553-4a0f-a997-e22b2ba0eda1" (UID: "e46a520c-3553-4a0f-a997-e22b2ba0eda1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:23:31.476309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.476210 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e46a520c-3553-4a0f-a997-e22b2ba0eda1" (UID: "e46a520c-3553-4a0f-a997-e22b2ba0eda1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:23:31.476309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.476233 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46a520c-3553-4a0f-a997-e22b2ba0eda1-kube-api-access-95n6k" (OuterVolumeSpecName: "kube-api-access-95n6k") pod "e46a520c-3553-4a0f-a997-e22b2ba0eda1" (UID: "e46a520c-3553-4a0f-a997-e22b2ba0eda1"). InnerVolumeSpecName "kube-api-access-95n6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:31.575637 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.575602 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:31.575637 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.575629 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e46a520c-3553-4a0f-a997-e22b2ba0eda1-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:31.575637 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:31.575639 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95n6k\" (UniqueName: \"kubernetes.io/projected/e46a520c-3553-4a0f-a997-e22b2ba0eda1-kube-api-access-95n6k\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:32.181369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.181333 2582 generic.go:356] "Generic (PLEG): container finished" podID="e46a520c-3553-4a0f-a997-e22b2ba0eda1" containerID="97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7" exitCode=0 Sep 17 18:23:32.181854 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.181380 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" Sep 17 18:23:32.181854 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.181403 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" event={"ID":"e46a520c-3553-4a0f-a997-e22b2ba0eda1","Type":"ContainerDied","Data":"97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7"} Sep 17 18:23:32.181854 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.181456 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd" event={"ID":"e46a520c-3553-4a0f-a997-e22b2ba0eda1","Type":"ContainerDied","Data":"c4efa8fe1989fdc2f1ee4fb6fa47bbd6b81205b85adb90a036cebc62690c2092"} Sep 17 18:23:32.181854 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.181490 2582 scope.go:117] "RemoveContainer" containerID="97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7" Sep 17 18:23:32.191740 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.191723 2582 scope.go:117] "RemoveContainer" containerID="97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7" Sep 17 18:23:32.192077 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:23:32.192057 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7\": container with ID starting with 97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7 not found: ID does not exist" containerID="97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7" Sep 17 18:23:32.192134 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.192086 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7"} err="failed to get container status \"97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7\": rpc error: code = NotFound desc = could not find container \"97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7\": container with ID starting with 97dab92202a5ebbb843597855d6653000d67a2a8791b337ba5173c1f5d627af7 not found: ID does not exist" Sep 17 18:23:32.204529 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.204504 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd"] Sep 17 18:23:32.208298 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:32.208273 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99ac10-77c8768d4f-fcbjd"] Sep 17 18:23:33.347669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:33.347629 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e46a520c-3553-4a0f-a997-e22b2ba0eda1" path="/var/lib/kubelet/pods/e46a520c-3553-4a0f-a997-e22b2ba0eda1/volumes" Sep 17 18:23:33.600904 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:33.600793 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:33.600904 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:33.600836 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:33.613492 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:33.613466 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:34.205132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:34.205102 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:23:43.800987 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:43.800946 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" podUID="be463b6f-4968-4ad7-bef2-f68f51c82025" containerName="main" containerID="cri-o://68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da" gracePeriod=45 Sep 17 18:23:44.068649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.068623 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:23:44.206501 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206471 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-dshm\") pod \"be463b6f-4968-4ad7-bef2-f68f51c82025\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " Sep 17 18:23:44.206680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206512 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk7f2\" (UniqueName: \"kubernetes.io/projected/be463b6f-4968-4ad7-bef2-f68f51c82025-kube-api-access-nk7f2\") pod \"be463b6f-4968-4ad7-bef2-f68f51c82025\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " Sep 17 18:23:44.206680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206572 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-tmp-dir\") pod \"be463b6f-4968-4ad7-bef2-f68f51c82025\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " Sep 17 18:23:44.206680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206594 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-home\") pod \"be463b6f-4968-4ad7-bef2-f68f51c82025\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " Sep 17 18:23:44.206680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206637 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-model-cache\") pod \"be463b6f-4968-4ad7-bef2-f68f51c82025\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " Sep 17 18:23:44.206680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206676 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be463b6f-4968-4ad7-bef2-f68f51c82025-tls-certs\") pod \"be463b6f-4968-4ad7-bef2-f68f51c82025\" (UID: \"be463b6f-4968-4ad7-bef2-f68f51c82025\") " Sep 17 18:23:44.207007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206789 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be463b6f-4968-4ad7-bef2-f68f51c82025" (UID: "be463b6f-4968-4ad7-bef2-f68f51c82025"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:44.207007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206930 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-model-cache" (OuterVolumeSpecName: "model-cache") pod "be463b6f-4968-4ad7-bef2-f68f51c82025" (UID: "be463b6f-4968-4ad7-bef2-f68f51c82025"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:44.207007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.206944 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-home" (OuterVolumeSpecName: "home") pod "be463b6f-4968-4ad7-bef2-f68f51c82025" (UID: "be463b6f-4968-4ad7-bef2-f68f51c82025"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:44.207190 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.207169 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:44.207258 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.207196 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:44.207258 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.207212 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:44.208754 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.208732 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be463b6f-4968-4ad7-bef2-f68f51c82025-kube-api-access-nk7f2" (OuterVolumeSpecName: "kube-api-access-nk7f2") pod "be463b6f-4968-4ad7-bef2-f68f51c82025" (UID: "be463b6f-4968-4ad7-bef2-f68f51c82025"). InnerVolumeSpecName "kube-api-access-nk7f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:23:44.209017 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.208997 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be463b6f-4968-4ad7-bef2-f68f51c82025-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be463b6f-4968-4ad7-bef2-f68f51c82025" (UID: "be463b6f-4968-4ad7-bef2-f68f51c82025"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:23:44.209112 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.209013 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-dshm" (OuterVolumeSpecName: "dshm") pod "be463b6f-4968-4ad7-bef2-f68f51c82025" (UID: "be463b6f-4968-4ad7-bef2-f68f51c82025"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:23:44.254425 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.254392 2582 generic.go:356] "Generic (PLEG): container finished" podID="be463b6f-4968-4ad7-bef2-f68f51c82025" containerID="68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da" exitCode=0 Sep 17 18:23:44.254552 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.254429 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" Sep 17 18:23:44.254552 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.254479 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" event={"ID":"be463b6f-4968-4ad7-bef2-f68f51c82025","Type":"ContainerDied","Data":"68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da"} Sep 17 18:23:44.254552 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.254521 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz" event={"ID":"be463b6f-4968-4ad7-bef2-f68f51c82025","Type":"ContainerDied","Data":"d89205e86b0312b34fe1e5bfe1e3ea0c1fbd87436d9e7c2774fe53d1420decf6"} Sep 17 18:23:44.254552 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.254538 2582 scope.go:117] "RemoveContainer" containerID="68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da" Sep 17 18:23:44.264072 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.264054 2582 scope.go:117] "RemoveContainer" containerID="68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da" Sep 17 18:23:44.264335 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:23:44.264315 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da\": container with ID starting with 68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da not found: ID does not exist" containerID="68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da" Sep 17 18:23:44.264379 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.264344 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da"} err="failed to get container status \"68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da\": rpc error: code = NotFound desc = could not find container \"68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da\": container with ID starting with 68e8e79072d8e84f542bed44035662f702375ff3466558d025d0d317066135da not found: ID does not exist" Sep 17 18:23:44.276529 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.276509 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz"] Sep 17 18:23:44.280718 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.280698 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-99ac10/test-model-99ac10-kserve-5b7ffbbcf-5kmgz"] Sep 17 18:23:44.307751 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.307695 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be463b6f-4968-4ad7-bef2-f68f51c82025-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:44.307751 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.307717 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk7f2\" (UniqueName: \"kubernetes.io/projected/be463b6f-4968-4ad7-bef2-f68f51c82025-kube-api-access-nk7f2\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:44.307751 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:44.307727 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be463b6f-4968-4ad7-bef2-f68f51c82025-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:23:45.347526 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:45.347487 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be463b6f-4968-4ad7-bef2-f68f51c82025" path="/var/lib/kubelet/pods/be463b6f-4968-4ad7-bef2-f68f51c82025/volumes" Sep 17 18:23:59.306184 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.306152 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr"] Sep 17 18:23:59.306744 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.306676 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e46a520c-3553-4a0f-a997-e22b2ba0eda1" containerName="maas-api" Sep 17 18:23:59.306744 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.306689 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e46a520c-3553-4a0f-a997-e22b2ba0eda1" containerName="maas-api" Sep 17 18:23:59.306744 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.306709 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be463b6f-4968-4ad7-bef2-f68f51c82025" containerName="main" Sep 17 18:23:59.306744 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.306718 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="be463b6f-4968-4ad7-bef2-f68f51c82025" containerName="main" Sep 17 18:23:59.307046 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.306858 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e46a520c-3553-4a0f-a997-e22b2ba0eda1" containerName="maas-api" Sep 17 18:23:59.307046 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.306905 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="be463b6f-4968-4ad7-bef2-f68f51c82025" containerName="main" Sep 17 18:23:59.310034 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.310013 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.313143 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.313120 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"default-dockercfg-t6hzs\"" Sep 17 18:23:59.313143 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.313139 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"distinct-2-w0-5d8053-kserve-self-signed-certs\"" Sep 17 18:23:59.313333 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.313163 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"openshift-service-ca.crt\"" Sep 17 18:23:59.313333 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.313211 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"kube-root-ca.crt\"" Sep 17 18:23:59.319004 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.318976 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr"] Sep 17 18:23:59.359155 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.359121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-home\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.359330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.359178 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcrpn\" (UniqueName: \"kubernetes.io/projected/5784409b-49f0-4286-80eb-baae97b50e91-kube-api-access-tcrpn\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.359330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.359266 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-dshm\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.359436 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.359336 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5784409b-49f0-4286-80eb-baae97b50e91-tls-certs\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.359436 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.359417 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-tmp-dir\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.359526 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.359451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-model-cache\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460266 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-tmp-dir\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460427 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-model-cache\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460427 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460345 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-home\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460427 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcrpn\" (UniqueName: \"kubernetes.io/projected/5784409b-49f0-4286-80eb-baae97b50e91-kube-api-access-tcrpn\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460427 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-dshm\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460501 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5784409b-49f0-4286-80eb-baae97b50e91-tls-certs\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-tmp-dir\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460743 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-model-cache\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.460929 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.460775 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-home\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.462798 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.462766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-dshm\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.463039 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.463021 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5784409b-49f0-4286-80eb-baae97b50e91-tls-certs\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.468329 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.468300 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcrpn\" (UniqueName: \"kubernetes.io/projected/5784409b-49f0-4286-80eb-baae97b50e91-kube-api-access-tcrpn\") pod \"distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.619769 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.619672 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:23:59.895700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:23:59.895672 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr"] Sep 17 18:23:59.896925 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:23:59.896898 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5784409b_49f0_4286_80eb_baae97b50e91.slice/crio-a510b7badc173ba2eafc8f792b55dd13b792847ef3da9755cca7d54d3f13e094 WatchSource:0}: Error finding container a510b7badc173ba2eafc8f792b55dd13b792847ef3da9755cca7d54d3f13e094: Status 404 returned error can't find the container with id a510b7badc173ba2eafc8f792b55dd13b792847ef3da9755cca7d54d3f13e094 Sep 17 18:24:00.363634 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:00.363600 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" event={"ID":"5784409b-49f0-4286-80eb-baae97b50e91","Type":"ContainerStarted","Data":"6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897"} Sep 17 18:24:00.364051 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:00.363644 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" event={"ID":"5784409b-49f0-4286-80eb-baae97b50e91","Type":"ContainerStarted","Data":"a510b7badc173ba2eafc8f792b55dd13b792847ef3da9755cca7d54d3f13e094"} Sep 17 18:24:00.381447 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:00.381402 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" podStartSLOduration=1.381387448 podStartE2EDuration="1.381387448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:00.379940328 +0000 UTC m=+1251.443884288" watchObservedRunningTime="2026-09-17 18:24:00.381387448 +0000 UTC m=+1251.445331418" Sep 17 18:24:09.619933 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:09.619852 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:24:09.619933 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:09.619946 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:24:09.632345 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:09.632319 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:24:10.431746 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:10.431717 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:24:22.720385 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.720343 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw"] Sep 17 18:24:22.725661 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.725637 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.728505 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.728479 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-95a92b-serving-cert\"" Sep 17 18:24:22.737493 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.737469 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw"] Sep 17 18:24:22.797546 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.797516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.797715 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.797554 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.797715 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.797574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7lcd\" (UniqueName: \"kubernetes.io/projected/54f1fca9-8852-4882-b0c3-9cac6d64604b-kube-api-access-r7lcd\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.797793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.797724 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f1fca9-8852-4882-b0c3-9cac6d64604b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.899001 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.898957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f1fca9-8852-4882-b0c3-9cac6d64604b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.899211 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.899015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.899211 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.899040 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.899211 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.899056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7lcd\" (UniqueName: \"kubernetes.io/projected/54f1fca9-8852-4882-b0c3-9cac6d64604b-kube-api-access-r7lcd\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.899703 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.899676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f1fca9-8852-4882-b0c3-9cac6d64604b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.901474 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.901454 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.901608 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.901585 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:22.908699 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:22.908676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7lcd\" (UniqueName: \"kubernetes.io/projected/54f1fca9-8852-4882-b0c3-9cac6d64604b-kube-api-access-r7lcd\") pod \"maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:23.040618 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:23.040529 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:23.318644 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:23.318618 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw"] Sep 17 18:24:23.321226 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:24:23.321194 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f1fca9_8852_4882_b0c3_9cac6d64604b.slice/crio-e718894a86dbfb74ff6254683bad7874fd295d667391a5f26c7681eef19ae807 WatchSource:0}: Error finding container e718894a86dbfb74ff6254683bad7874fd295d667391a5f26c7681eef19ae807: Status 404 returned error can't find the container with id e718894a86dbfb74ff6254683bad7874fd295d667391a5f26c7681eef19ae807 Sep 17 18:24:23.500283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:23.500242 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" event={"ID":"54f1fca9-8852-4882-b0c3-9cac6d64604b","Type":"ContainerStarted","Data":"e718894a86dbfb74ff6254683bad7874fd295d667391a5f26c7681eef19ae807"} Sep 17 18:24:24.507209 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:24.507169 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" event={"ID":"54f1fca9-8852-4882-b0c3-9cac6d64604b","Type":"ContainerStarted","Data":"6ac2e3059fd4c71f7d609d85222edabc17b489022f547d589c33e71f8c8b217e"} Sep 17 18:24:24.507690 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:24.507260 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:24.528826 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:24.528776 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" podStartSLOduration=2.013399383 podStartE2EDuration="2.528760844s" totalImagesPullingTime="515.361461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:24.52709576 +0000 UTC m=+1275.591039734" watchObservedRunningTime="2026-09-17 18:24:24.528760844 +0000 UTC m=+1275.592704803" Sep 17 18:24:30.518608 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:30.518577 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:24:57.437684 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.437627 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p"] Sep 17 18:24:57.442335 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.442313 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.445461 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.445433 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-d81d7c-serving-cert\"" Sep 17 18:24:57.454572 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.454546 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p"] Sep 17 18:24:57.466986 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.466950 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mcp\" (UniqueName: \"kubernetes.io/projected/03f25ae0-cbc6-423a-b022-ed931bfc483e-kube-api-access-69mcp\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.467135 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.467113 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.467181 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.467151 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f25ae0-cbc6-423a-b022-ed931bfc483e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.467265 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.467249 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.568243 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.568216 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.568408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.568265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mcp\" (UniqueName: \"kubernetes.io/projected/03f25ae0-cbc6-423a-b022-ed931bfc483e-kube-api-access-69mcp\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.568408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.568359 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.568408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.568399 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f25ae0-cbc6-423a-b022-ed931bfc483e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.568517 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:24:57.568454 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-serving-cert: secret "maas-api-e2e-shared-b-w3-d81d7c-serving-cert" not found Sep 17 18:24:57.568552 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:24:57.568531 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls podName:03f25ae0-cbc6-423a-b022-ed931bfc483e nodeName:}" failed. No retries permitted until 2026-09-17 18:24:58.068513465 +0000 UTC m=+1309.132457401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls") pod "maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" (UID: "03f25ae0-cbc6-423a-b022-ed931bfc483e") : secret "maas-api-e2e-shared-b-w3-d81d7c-serving-cert" not found Sep 17 18:24:57.569054 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.569031 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f25ae0-cbc6-423a-b022-ed931bfc483e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.570640 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.570613 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:57.576804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:57.576781 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mcp\" (UniqueName: \"kubernetes.io/projected/03f25ae0-cbc6-423a-b022-ed931bfc483e-kube-api-access-69mcp\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:58.074387 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:58.074350 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:58.076808 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:58.076787 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:58.357802 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:58.357713 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:58.633710 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:58.633684 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p"] Sep 17 18:24:58.636109 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:24:58.636083 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f25ae0_cbc6_423a_b022_ed931bfc483e.slice/crio-dca44555d14db632202086fad153531da50f9e10c48ed1cc606775b1507bc429 WatchSource:0}: Error finding container dca44555d14db632202086fad153531da50f9e10c48ed1cc606775b1507bc429: Status 404 returned error can't find the container with id dca44555d14db632202086fad153531da50f9e10c48ed1cc606775b1507bc429 Sep 17 18:24:58.637905 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:58.637850 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:24:58.715759 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:58.715729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" event={"ID":"03f25ae0-cbc6-423a-b022-ed931bfc483e","Type":"ContainerStarted","Data":"dca44555d14db632202086fad153531da50f9e10c48ed1cc606775b1507bc429"} Sep 17 18:24:59.723827 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:59.723791 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" event={"ID":"03f25ae0-cbc6-423a-b022-ed931bfc483e","Type":"ContainerStarted","Data":"cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3"} Sep 17 18:24:59.724291 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:59.723983 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:24:59.743997 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:24:59.743943 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" podStartSLOduration=2.381804873 podStartE2EDuration="2.743929148s" totalImagesPullingTime="362.124275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:59.741962811 +0000 UTC m=+1310.805906771" watchObservedRunningTime="2026-09-17 18:24:59.743929148 +0000 UTC m=+1310.807873102" Sep 17 18:25:05.734751 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:05.734712 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:25:12.704925 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:12.704888 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294"] Sep 17 18:25:27.725321 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.725203 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" podUID="d86461fe-234f-4bab-a7bd-2dceb03f504d" containerName="main" containerID="cri-o://574d5d6bb42bc7f0d297ad1f9d4f41799786ad6100b987ea91592a2c05c8ab33" gracePeriod=45 Sep 17 18:25:27.892921 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.892873 2582 generic.go:356] "Generic (PLEG): container finished" podID="d86461fe-234f-4bab-a7bd-2dceb03f504d" containerID="574d5d6bb42bc7f0d297ad1f9d4f41799786ad6100b987ea91592a2c05c8ab33" exitCode=0 Sep 17 18:25:27.893098 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.892927 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" event={"ID":"d86461fe-234f-4bab-a7bd-2dceb03f504d","Type":"ContainerDied","Data":"574d5d6bb42bc7f0d297ad1f9d4f41799786ad6100b987ea91592a2c05c8ab33"} Sep 17 18:25:27.986256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986192 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:25:27.986625 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986609 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-tmp-dir\") pod \"d86461fe-234f-4bab-a7bd-2dceb03f504d\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " Sep 17 18:25:27.986702 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986649 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86461fe-234f-4bab-a7bd-2dceb03f504d-tls-certs\") pod \"d86461fe-234f-4bab-a7bd-2dceb03f504d\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " Sep 17 18:25:27.986702 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986670 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-home\") pod \"d86461fe-234f-4bab-a7bd-2dceb03f504d\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " Sep 17 18:25:27.986702 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986700 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-model-cache\") pod \"d86461fe-234f-4bab-a7bd-2dceb03f504d\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " Sep 17 18:25:27.986877 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986717 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dd4z\" (UniqueName: \"kubernetes.io/projected/d86461fe-234f-4bab-a7bd-2dceb03f504d-kube-api-access-8dd4z\") pod \"d86461fe-234f-4bab-a7bd-2dceb03f504d\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " Sep 17 18:25:27.986877 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986811 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-dshm\") pod \"d86461fe-234f-4bab-a7bd-2dceb03f504d\" (UID: \"d86461fe-234f-4bab-a7bd-2dceb03f504d\") " Sep 17 18:25:27.986995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986958 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-home" (OuterVolumeSpecName: "home") pod "d86461fe-234f-4bab-a7bd-2dceb03f504d" (UID: "d86461fe-234f-4bab-a7bd-2dceb03f504d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:25:27.987042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986988 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d86461fe-234f-4bab-a7bd-2dceb03f504d" (UID: "d86461fe-234f-4bab-a7bd-2dceb03f504d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:25:27.987042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.986940 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d86461fe-234f-4bab-a7bd-2dceb03f504d" (UID: "d86461fe-234f-4bab-a7bd-2dceb03f504d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:25:27.987294 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.987272 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:25:27.987371 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.987297 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:25:27.987371 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.987310 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:25:27.988946 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.988916 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-dshm" (OuterVolumeSpecName: "dshm") pod "d86461fe-234f-4bab-a7bd-2dceb03f504d" (UID: "d86461fe-234f-4bab-a7bd-2dceb03f504d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:25:27.989126 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.989105 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86461fe-234f-4bab-a7bd-2dceb03f504d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d86461fe-234f-4bab-a7bd-2dceb03f504d" (UID: "d86461fe-234f-4bab-a7bd-2dceb03f504d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:25:27.989196 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:27.989151 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86461fe-234f-4bab-a7bd-2dceb03f504d-kube-api-access-8dd4z" (OuterVolumeSpecName: "kube-api-access-8dd4z") pod "d86461fe-234f-4bab-a7bd-2dceb03f504d" (UID: "d86461fe-234f-4bab-a7bd-2dceb03f504d"). InnerVolumeSpecName "kube-api-access-8dd4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:25:28.088035 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:28.088005 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86461fe-234f-4bab-a7bd-2dceb03f504d-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:25:28.088035 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:28.088032 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86461fe-234f-4bab-a7bd-2dceb03f504d-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:25:28.088035 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:28.088043 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dd4z\" (UniqueName: \"kubernetes.io/projected/d86461fe-234f-4bab-a7bd-2dceb03f504d-kube-api-access-8dd4z\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:25:28.899641 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:28.899608 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" Sep 17 18:25:28.900059 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:28.899649 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294" event={"ID":"d86461fe-234f-4bab-a7bd-2dceb03f504d","Type":"ContainerDied","Data":"afc6e6b1e6dfebe11874e91d2e79739bacd310bcaf18f557f4fd8dcb573fdbc7"} Sep 17 18:25:28.900059 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:28.899703 2582 scope.go:117] "RemoveContainer" containerID="574d5d6bb42bc7f0d297ad1f9d4f41799786ad6100b987ea91592a2c05c8ab33" Sep 17 18:25:28.922583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:28.922558 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294"] Sep 17 18:25:28.928208 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:28.928184 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-b-92400e60-kserve-84f99957fc-5c294"] Sep 17 18:25:29.346819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:29.346736 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86461fe-234f-4bab-a7bd-2dceb03f504d" path="/var/lib/kubelet/pods/d86461fe-234f-4bab-a7bd-2dceb03f504d/volumes" Sep 17 18:25:32.182379 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.182339 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb"] Sep 17 18:25:32.183352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.183302 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86461fe-234f-4bab-a7bd-2dceb03f504d" containerName="main" Sep 17 18:25:32.183352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.183326 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86461fe-234f-4bab-a7bd-2dceb03f504d" containerName="main" Sep 17 18:25:32.183700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.183603 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86461fe-234f-4bab-a7bd-2dceb03f504d" containerName="main" Sep 17 18:25:32.187696 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.187675 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.190584 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.190546 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"distinct-w4-b18433-kserve-self-signed-certs\"" Sep 17 18:25:32.197544 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.197516 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb"] Sep 17 18:25:32.224084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.224052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-model-cache\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.224234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.224090 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-dshm\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.224234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.224126 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-home\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.224234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.224199 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-tmp-dir\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.224354 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.224267 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4z8b\" (UniqueName: \"kubernetes.io/projected/4a507a93-f7b6-4d23-944d-70c47230218a-kube-api-access-b4z8b\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.224354 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.224302 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a507a93-f7b6-4d23-944d-70c47230218a-tls-certs\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.325468 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.325429 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-model-cache\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.325649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.325474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-dshm\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.325649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.325521 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-home\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.325649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.325565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-tmp-dir\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.325649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.325639 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4z8b\" (UniqueName: \"kubernetes.io/projected/4a507a93-f7b6-4d23-944d-70c47230218a-kube-api-access-b4z8b\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.325854 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.325678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a507a93-f7b6-4d23-944d-70c47230218a-tls-certs\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.325952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.325914 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-model-cache\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.325952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.325943 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-home\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.326050 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.326026 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-tmp-dir\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.327900 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.327847 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-dshm\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.328404 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.328389 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a507a93-f7b6-4d23-944d-70c47230218a-tls-certs\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.335585 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.335563 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4z8b\" (UniqueName: \"kubernetes.io/projected/4a507a93-f7b6-4d23-944d-70c47230218a-kube-api-access-b4z8b\") pod \"distinct-w4-b18433-kserve-5768fc7b56-qvtcb\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.499254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.499176 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:32.773662 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.773637 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb"] Sep 17 18:25:32.775560 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:25:32.775529 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a507a93_f7b6_4d23_944d_70c47230218a.slice/crio-6331facf596dd5533704fc885bebb510fe94686a86d25ffb2a352e3552fe8384 WatchSource:0}: Error finding container 6331facf596dd5533704fc885bebb510fe94686a86d25ffb2a352e3552fe8384: Status 404 returned error can't find the container with id 6331facf596dd5533704fc885bebb510fe94686a86d25ffb2a352e3552fe8384 Sep 17 18:25:32.924497 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.924464 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" event={"ID":"4a507a93-f7b6-4d23-944d-70c47230218a","Type":"ContainerStarted","Data":"280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4"} Sep 17 18:25:32.924497 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.924506 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" event={"ID":"4a507a93-f7b6-4d23-944d-70c47230218a","Type":"ContainerStarted","Data":"6331facf596dd5533704fc885bebb510fe94686a86d25ffb2a352e3552fe8384"} Sep 17 18:25:32.943599 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:32.943548 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" podStartSLOduration=0.943533556 podStartE2EDuration="943.533556ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:32.940326471 +0000 UTC m=+1344.004270428" watchObservedRunningTime="2026-09-17 18:25:32.943533556 +0000 UTC m=+1344.007477514" Sep 17 18:25:41.499168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.499111 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph"] Sep 17 18:25:41.503439 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.503415 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.505848 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.505827 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"kube-root-ca.crt\"" Sep 17 18:25:41.506627 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.506608 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"auth-test-model-95a92b-kserve-self-signed-certs\"" Sep 17 18:25:41.506724 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.506666 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"openshift-service-ca.crt\"" Sep 17 18:25:41.506724 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.506672 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"default-dockercfg-rf2c5\"" Sep 17 18:25:41.512310 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.512286 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph"] Sep 17 18:25:41.617445 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.617411 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-model-cache\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.617445 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.617450 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6q4n\" (UniqueName: \"kubernetes.io/projected/bf805383-407b-4dcc-850e-c1933724a004-kube-api-access-m6q4n\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.617712 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.617627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-dshm\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.617712 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.617674 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf805383-407b-4dcc-850e-c1933724a004-tls-certs\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.617831 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.617737 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-home\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.617831 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.617796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-tmp-dir\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719193 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719161 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-tmp-dir\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719386 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-model-cache\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719386 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719252 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6q4n\" (UniqueName: \"kubernetes.io/projected/bf805383-407b-4dcc-850e-c1933724a004-kube-api-access-m6q4n\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719386 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719316 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-dshm\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719386 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719336 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf805383-407b-4dcc-850e-c1933724a004-tls-certs\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719386 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-home\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719649 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-tmp-dir\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719678 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719668 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-model-cache\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.719784 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.719757 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-home\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.721751 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.721725 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-dshm\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.721972 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.721957 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf805383-407b-4dcc-850e-c1933724a004-tls-certs\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.728118 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.728094 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6q4n\" (UniqueName: \"kubernetes.io/projected/bf805383-407b-4dcc-850e-c1933724a004-kube-api-access-m6q4n\") pod \"auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:41.817158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:41.817085 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:42.096693 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:42.096602 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph"] Sep 17 18:25:42.103393 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:25:42.103367 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf805383_407b_4dcc_850e_c1933724a004.slice/crio-1c8cb6873f445b7151a12b341d29bc35e22482b90194ec8e856ec00884abf708 WatchSource:0}: Error finding container 1c8cb6873f445b7151a12b341d29bc35e22482b90194ec8e856ec00884abf708: Status 404 returned error can't find the container with id 1c8cb6873f445b7151a12b341d29bc35e22482b90194ec8e856ec00884abf708 Sep 17 18:25:42.499695 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:42.499656 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:42.500108 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:42.499714 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:42.512252 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:42.512228 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:42.987192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:42.987154 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" event={"ID":"bf805383-407b-4dcc-850e-c1933724a004","Type":"ContainerStarted","Data":"11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1"} Sep 17 18:25:42.987192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:42.987202 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" event={"ID":"bf805383-407b-4dcc-850e-c1933724a004","Type":"ContainerStarted","Data":"1c8cb6873f445b7151a12b341d29bc35e22482b90194ec8e856ec00884abf708"} Sep 17 18:25:42.998574 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:42.998550 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:25:43.007784 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:43.007739 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" podStartSLOduration=2.007725426 podStartE2EDuration="2.007725426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:43.005329822 +0000 UTC m=+1354.069273781" watchObservedRunningTime="2026-09-17 18:25:43.007725426 +0000 UTC m=+1354.071669384" Sep 17 18:25:51.817732 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:51.817694 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:51.818180 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:51.817759 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:51.830970 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:51.830948 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:25:52.058215 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:25:52.058186 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:26:06.572178 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.572144 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v"] Sep 17 18:26:06.572652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.572350 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" podUID="9be6fab0-c4ad-483f-8f6b-b127f8b04615" containerName="maas-api" containerID="cri-o://d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3" gracePeriod=30 Sep 17 18:26:06.837560 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.837502 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:26:06.884000 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.883941 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-maas-api-tls\") pod \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " Sep 17 18:26:06.884261 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.884243 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-metrics-certs\") pod \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " Sep 17 18:26:06.884387 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.884371 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79xcl\" (UniqueName: \"kubernetes.io/projected/9be6fab0-c4ad-483f-8f6b-b127f8b04615-kube-api-access-79xcl\") pod \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " Sep 17 18:26:06.884854 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.884826 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be6fab0-c4ad-483f-8f6b-b127f8b04615-openshift-service-ca\") pod \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\" (UID: \"9be6fab0-c4ad-483f-8f6b-b127f8b04615\") " Sep 17 18:26:06.885508 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.885188 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be6fab0-c4ad-483f-8f6b-b127f8b04615-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9be6fab0-c4ad-483f-8f6b-b127f8b04615" (UID: "9be6fab0-c4ad-483f-8f6b-b127f8b04615"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:26:06.885508 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.885488 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9be6fab0-c4ad-483f-8f6b-b127f8b04615-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:26:06.886921 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.886814 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be6fab0-c4ad-483f-8f6b-b127f8b04615-kube-api-access-79xcl" (OuterVolumeSpecName: "kube-api-access-79xcl") pod "9be6fab0-c4ad-483f-8f6b-b127f8b04615" (UID: "9be6fab0-c4ad-483f-8f6b-b127f8b04615"). InnerVolumeSpecName "kube-api-access-79xcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:26:06.887045 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.886970 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9be6fab0-c4ad-483f-8f6b-b127f8b04615" (UID: "9be6fab0-c4ad-483f-8f6b-b127f8b04615"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:26:06.887121 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.887070 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9be6fab0-c4ad-483f-8f6b-b127f8b04615" (UID: "9be6fab0-c4ad-483f-8f6b-b127f8b04615"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:26:06.986549 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.986515 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:26:06.986549 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.986542 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9be6fab0-c4ad-483f-8f6b-b127f8b04615-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:26:06.986549 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:06.986551 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79xcl\" (UniqueName: \"kubernetes.io/projected/9be6fab0-c4ad-483f-8f6b-b127f8b04615-kube-api-access-79xcl\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:26:07.135627 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.135540 2582 generic.go:356] "Generic (PLEG): container finished" podID="9be6fab0-c4ad-483f-8f6b-b127f8b04615" containerID="d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3" exitCode=0 Sep 17 18:26:07.135627 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.135579 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" Sep 17 18:26:07.135627 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.135607 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" event={"ID":"9be6fab0-c4ad-483f-8f6b-b127f8b04615","Type":"ContainerDied","Data":"d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3"} Sep 17 18:26:07.135889 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.135656 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v" event={"ID":"9be6fab0-c4ad-483f-8f6b-b127f8b04615","Type":"ContainerDied","Data":"d316a291ec3f0bcfeb1a29b686d906dd55b94b18f3f1d4d97ba5e877d9c0d791"} Sep 17 18:26:07.135889 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.135683 2582 scope.go:117] "RemoveContainer" containerID="d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3" Sep 17 18:26:07.148430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.148407 2582 scope.go:117] "RemoveContainer" containerID="d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3" Sep 17 18:26:07.148732 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:26:07.148712 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3\": container with ID starting with d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3 not found: ID does not exist" containerID="d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3" Sep 17 18:26:07.148797 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.148748 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3"} err="failed to get container status \"d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3\": rpc error: code = NotFound desc = could not find container \"d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3\": container with ID starting with d1d0630ea9fa5e5a54a008341d26c5b563ecb8451c8020bbd211c48864f0c6f3 not found: ID does not exist" Sep 17 18:26:07.163634 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.163607 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v"] Sep 17 18:26:07.167606 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.167580 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d245d137-6b84f4dbbc-q5k2v"] Sep 17 18:26:07.347813 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:26:07.347783 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9be6fab0-c4ad-483f-8f6b-b127f8b04615" path="/var/lib/kubelet/pods/9be6fab0-c4ad-483f-8f6b-b127f8b04615/volumes" Sep 17 18:28:11.370054 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:11.370029 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:28:11.373231 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:11.373212 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:28:11.385268 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:11.385247 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:28:11.388713 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:11.388694 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:28:38.299105 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.299025 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn"] Sep 17 18:28:38.299571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.299515 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be6fab0-c4ad-483f-8f6b-b127f8b04615" containerName="maas-api" Sep 17 18:28:38.299571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.299527 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be6fab0-c4ad-483f-8f6b-b127f8b04615" containerName="maas-api" Sep 17 18:28:38.299646 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.299601 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be6fab0-c4ad-483f-8f6b-b127f8b04615" containerName="maas-api" Sep 17 18:28:38.302819 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.302803 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.306707 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.306679 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4faba0-serving-cert\"" Sep 17 18:28:38.330126 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.330094 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn"] Sep 17 18:28:38.470719 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.470680 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8e356e-3830-4d95-a1c3-1aa456371a17-openshift-service-ca\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.470719 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.470723 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw5kr\" (UniqueName: \"kubernetes.io/projected/5a8e356e-3830-4d95-a1c3-1aa456371a17-kube-api-access-bw5kr\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.470965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.470747 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-maas-api-tls\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.470965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.470888 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-metrics-certs\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.572401 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.572322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8e356e-3830-4d95-a1c3-1aa456371a17-openshift-service-ca\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.572401 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.572364 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw5kr\" (UniqueName: \"kubernetes.io/projected/5a8e356e-3830-4d95-a1c3-1aa456371a17-kube-api-access-bw5kr\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.572593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.572406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-maas-api-tls\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.572593 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.572489 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-metrics-certs\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.573015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.572991 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8e356e-3830-4d95-a1c3-1aa456371a17-openshift-service-ca\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.574716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.574692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-maas-api-tls\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.574967 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.574949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-metrics-certs\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.581136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.581115 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw5kr\" (UniqueName: \"kubernetes.io/projected/5a8e356e-3830-4d95-a1c3-1aa456371a17-kube-api-access-bw5kr\") pod \"maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.613212 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.613181 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:38.895513 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:38.895485 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn"] Sep 17 18:28:38.897838 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:28:38.897802 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8e356e_3830_4d95_a1c3_1aa456371a17.slice/crio-59b4fb2d301b577810f36474c947445de4ceb73d2bf9f05ebf577ec706386578 WatchSource:0}: Error finding container 59b4fb2d301b577810f36474c947445de4ceb73d2bf9f05ebf577ec706386578: Status 404 returned error can't find the container with id 59b4fb2d301b577810f36474c947445de4ceb73d2bf9f05ebf577ec706386578 Sep 17 18:28:39.040688 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:39.040650 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" event={"ID":"5a8e356e-3830-4d95-a1c3-1aa456371a17","Type":"ContainerStarted","Data":"59b4fb2d301b577810f36474c947445de4ceb73d2bf9f05ebf577ec706386578"} Sep 17 18:28:40.048926 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:40.048886 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" event={"ID":"5a8e356e-3830-4d95-a1c3-1aa456371a17","Type":"ContainerStarted","Data":"654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038"} Sep 17 18:28:40.049357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:40.048960 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:28:40.068336 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:40.068290 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" podStartSLOduration=1.501348522 podStartE2EDuration="2.068273797s" totalImagesPullingTime="566.925275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:40.065832301 +0000 UTC m=+1531.129776260" watchObservedRunningTime="2026-09-17 18:28:40.068273797 +0000 UTC m=+1531.132217754" Sep 17 18:28:46.060142 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:28:46.060112 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:29:12.514054 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.514024 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v"] Sep 17 18:29:12.518084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.518062 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.520287 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.520259 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b06908-serving-cert\"" Sep 17 18:29:12.529023 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.528997 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v"] Sep 17 18:29:12.627557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.627524 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtbtp\" (UniqueName: \"kubernetes.io/projected/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-kube-api-access-mtbtp\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.627746 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.627572 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.627746 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.627675 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-maas-api-tls\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.627912 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.627759 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-metrics-certs\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.728847 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.728818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-metrics-certs\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.729037 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.728940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtbtp\" (UniqueName: \"kubernetes.io/projected/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-kube-api-access-mtbtp\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.729037 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.728971 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.729037 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.729011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-maas-api-tls\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.729683 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.729657 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.731431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.731404 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-maas-api-tls\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.731523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.731442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-metrics-certs\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.737259 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.737239 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtbtp\" (UniqueName: \"kubernetes.io/projected/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-kube-api-access-mtbtp\") pod \"maas-api-e2e-api-b-b06908-7985d99b86-rg57v\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:12.830458 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:12.830365 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:13.126890 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:13.126842 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v"] Sep 17 18:29:13.127602 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:29:13.127575 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec9d9b8_cb8d_4d7b_b615_aa50618db1f2.slice/crio-8ca7969b8eb3353c5e700fde28378bc9b0980939ad78612ad9c6aa3d0578e99b WatchSource:0}: Error finding container 8ca7969b8eb3353c5e700fde28378bc9b0980939ad78612ad9c6aa3d0578e99b: Status 404 returned error can't find the container with id 8ca7969b8eb3353c5e700fde28378bc9b0980939ad78612ad9c6aa3d0578e99b Sep 17 18:29:13.253309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:13.253261 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" event={"ID":"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2","Type":"ContainerStarted","Data":"8ca7969b8eb3353c5e700fde28378bc9b0980939ad78612ad9c6aa3d0578e99b"} Sep 17 18:29:14.261851 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:14.261813 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" event={"ID":"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2","Type":"ContainerStarted","Data":"36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c"} Sep 17 18:29:14.262331 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:14.261897 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:14.282330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:14.282279 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" podStartSLOduration=1.9244964420000001 podStartE2EDuration="2.282263941s" totalImagesPullingTime="357.767499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:14.279836781 +0000 UTC m=+1565.343780741" watchObservedRunningTime="2026-09-17 18:29:14.282263941 +0000 UTC m=+1565.346207899" Sep 17 18:29:20.272987 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:20.272955 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:29:42.957811 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:42.957780 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb"] Sep 17 18:29:42.969570 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:42.969542 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p"] Sep 17 18:29:53.868236 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:53.868197 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4"] Sep 17 18:29:53.868803 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:53.868444 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" podUID="e56258a1-d9fa-43ee-8495-88e71277f744" containerName="maas-api" containerID="cri-o://7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f" gracePeriod=30 Sep 17 18:29:53.928304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:53.928268 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm"] Sep 17 18:29:53.928571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:53.928541 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" podUID="9b42d62b-e740-4a13-8864-784ad5829e6a" containerName="payload-processing" containerID="cri-o://e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779" gracePeriod=30 Sep 17 18:29:54.202406 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.202377 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:29:54.236826 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.236796 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:29:54.276107 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.276079 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-metrics-certs\") pod \"e56258a1-d9fa-43ee-8495-88e71277f744\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " Sep 17 18:29:54.276307 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.276153 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-maas-api-tls\") pod \"e56258a1-d9fa-43ee-8495-88e71277f744\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " Sep 17 18:29:54.276307 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.276261 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56258a1-d9fa-43ee-8495-88e71277f744-openshift-service-ca\") pod \"e56258a1-d9fa-43ee-8495-88e71277f744\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " Sep 17 18:29:54.276433 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.276350 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxts9\" (UniqueName: \"kubernetes.io/projected/e56258a1-d9fa-43ee-8495-88e71277f744-kube-api-access-zxts9\") pod \"e56258a1-d9fa-43ee-8495-88e71277f744\" (UID: \"e56258a1-d9fa-43ee-8495-88e71277f744\") " Sep 17 18:29:54.276693 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.276663 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56258a1-d9fa-43ee-8495-88e71277f744-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e56258a1-d9fa-43ee-8495-88e71277f744" (UID: "e56258a1-d9fa-43ee-8495-88e71277f744"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:29:54.276831 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.276812 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e56258a1-d9fa-43ee-8495-88e71277f744-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:54.278328 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.278303 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e56258a1-d9fa-43ee-8495-88e71277f744" (UID: "e56258a1-d9fa-43ee-8495-88e71277f744"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:29:54.278722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.278702 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e56258a1-d9fa-43ee-8495-88e71277f744" (UID: "e56258a1-d9fa-43ee-8495-88e71277f744"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:29:54.278798 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.278721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56258a1-d9fa-43ee-8495-88e71277f744-kube-api-access-zxts9" (OuterVolumeSpecName: "kube-api-access-zxts9") pod "e56258a1-d9fa-43ee-8495-88e71277f744" (UID: "e56258a1-d9fa-43ee-8495-88e71277f744"). InnerVolumeSpecName "kube-api-access-zxts9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:29:54.377366 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.377336 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b42d62b-e740-4a13-8864-784ad5829e6a-plugins-config-volume\") pod \"9b42d62b-e740-4a13-8864-784ad5829e6a\" (UID: \"9b42d62b-e740-4a13-8864-784ad5829e6a\") " Sep 17 18:29:54.377535 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.377399 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbtfc\" (UniqueName: \"kubernetes.io/projected/9b42d62b-e740-4a13-8864-784ad5829e6a-kube-api-access-bbtfc\") pod \"9b42d62b-e740-4a13-8864-784ad5829e6a\" (UID: \"9b42d62b-e740-4a13-8864-784ad5829e6a\") " Sep 17 18:29:54.377710 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.377688 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b42d62b-e740-4a13-8864-784ad5829e6a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b42d62b-e740-4a13-8864-784ad5829e6a" (UID: "9b42d62b-e740-4a13-8864-784ad5829e6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:29:54.377757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.377705 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:54.377757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.377727 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxts9\" (UniqueName: \"kubernetes.io/projected/e56258a1-d9fa-43ee-8495-88e71277f744-kube-api-access-zxts9\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:54.377757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.377743 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e56258a1-d9fa-43ee-8495-88e71277f744-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:54.379430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.379413 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b42d62b-e740-4a13-8864-784ad5829e6a-kube-api-access-bbtfc" (OuterVolumeSpecName: "kube-api-access-bbtfc") pod "9b42d62b-e740-4a13-8864-784ad5829e6a" (UID: "9b42d62b-e740-4a13-8864-784ad5829e6a"). InnerVolumeSpecName "kube-api-access-bbtfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:29:54.478484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.478405 2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b42d62b-e740-4a13-8864-784ad5829e6a-plugins-config-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:54.478484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.478442 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbtfc\" (UniqueName: \"kubernetes.io/projected/9b42d62b-e740-4a13-8864-784ad5829e6a-kube-api-access-bbtfc\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:54.516590 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.516563 2582 generic.go:356] "Generic (PLEG): container finished" podID="9b42d62b-e740-4a13-8864-784ad5829e6a" containerID="e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779" exitCode=0 Sep 17 18:29:54.516736 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.516608 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" Sep 17 18:29:54.516736 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.516658 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" event={"ID":"9b42d62b-e740-4a13-8864-784ad5829e6a","Type":"ContainerDied","Data":"e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779"} Sep 17 18:29:54.516736 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.516718 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm" event={"ID":"9b42d62b-e740-4a13-8864-784ad5829e6a","Type":"ContainerDied","Data":"82531838fe091f986df4571df6f657654a3e6d3aaf4b8c13555382009b9c11df"} Sep 17 18:29:54.516904 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.516755 2582 scope.go:117] "RemoveContainer" containerID="e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779" Sep 17 18:29:54.518132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.518111 2582 generic.go:356] "Generic (PLEG): container finished" podID="e56258a1-d9fa-43ee-8495-88e71277f744" containerID="7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f" exitCode=0 Sep 17 18:29:54.518200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.518147 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" Sep 17 18:29:54.518200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.518151 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" event={"ID":"e56258a1-d9fa-43ee-8495-88e71277f744","Type":"ContainerDied","Data":"7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f"} Sep 17 18:29:54.518200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.518194 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4" event={"ID":"e56258a1-d9fa-43ee-8495-88e71277f744","Type":"ContainerDied","Data":"630898fb038cab5d60d301b3ad7d92a71ba7a810f34b4553d4098b43b747f4c9"} Sep 17 18:29:54.527052 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.527033 2582 scope.go:117] "RemoveContainer" containerID="e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779" Sep 17 18:29:54.527406 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:29:54.527384 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779\": container with ID starting with e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779 not found: ID does not exist" containerID="e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779" Sep 17 18:29:54.527480 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.527416 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779"} err="failed to get container status \"e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779\": rpc error: code = NotFound desc = could not find container \"e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779\": container with ID starting with e72c15d8f4b048ce2d6616251a262003efad69515c049e4c11f0fe9f15101779 not found: ID does not exist" Sep 17 18:29:54.527480 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.527434 2582 scope.go:117] "RemoveContainer" containerID="7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f" Sep 17 18:29:54.535788 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.535768 2582 scope.go:117] "RemoveContainer" containerID="7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f" Sep 17 18:29:54.536064 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:29:54.536046 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f\": container with ID starting with 7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f not found: ID does not exist" containerID="7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f" Sep 17 18:29:54.536137 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.536075 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f"} err="failed to get container status \"7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f\": rpc error: code = NotFound desc = could not find container \"7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f\": container with ID starting with 7b0f4b895a08a4af44143759d5a19c450532dd9a7a3926c5abb6c7451ad3791f not found: ID does not exist" Sep 17 18:29:54.543020 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.542998 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm"] Sep 17 18:29:54.546841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.546815 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-b18433-69b8b6ff7c-fkzwm"] Sep 17 18:29:54.556180 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.556159 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4"] Sep 17 18:29:54.558496 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:54.558477 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b18433-84b9b48bd7-ztfm4"] Sep 17 18:29:55.219485 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.219448 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn"] Sep 17 18:29:55.220010 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.219779 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" podUID="5a8e356e-3830-4d95-a1c3-1aa456371a17" containerName="maas-api" containerID="cri-o://654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038" gracePeriod=30 Sep 17 18:29:55.348681 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.348637 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b42d62b-e740-4a13-8864-784ad5829e6a" path="/var/lib/kubelet/pods/9b42d62b-e740-4a13-8864-784ad5829e6a/volumes" Sep 17 18:29:55.349443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.349393 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e56258a1-d9fa-43ee-8495-88e71277f744" path="/var/lib/kubelet/pods/e56258a1-d9fa-43ee-8495-88e71277f744/volumes" Sep 17 18:29:55.497170 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.497144 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:29:55.527152 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.527112 2582 generic.go:356] "Generic (PLEG): container finished" podID="5a8e356e-3830-4d95-a1c3-1aa456371a17" containerID="654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038" exitCode=0 Sep 17 18:29:55.527326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.527166 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" Sep 17 18:29:55.527326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.527167 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" event={"ID":"5a8e356e-3830-4d95-a1c3-1aa456371a17","Type":"ContainerDied","Data":"654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038"} Sep 17 18:29:55.527326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.527233 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn" event={"ID":"5a8e356e-3830-4d95-a1c3-1aa456371a17","Type":"ContainerDied","Data":"59b4fb2d301b577810f36474c947445de4ceb73d2bf9f05ebf577ec706386578"} Sep 17 18:29:55.527326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.527251 2582 scope.go:117] "RemoveContainer" containerID="654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038" Sep 17 18:29:55.543592 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.543568 2582 scope.go:117] "RemoveContainer" containerID="654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038" Sep 17 18:29:55.544017 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:29:55.543986 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038\": container with ID starting with 654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038 not found: ID does not exist" containerID="654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038" Sep 17 18:29:55.544120 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.544026 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038"} err="failed to get container status \"654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038\": rpc error: code = NotFound desc = could not find container \"654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038\": container with ID starting with 654b3dfc9e1779d194d27d0921513bf868fe4eea5b2efb3cfbfe576b3c2a0038 not found: ID does not exist" Sep 17 18:29:55.590126 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.590097 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-metrics-certs\") pod \"5a8e356e-3830-4d95-a1c3-1aa456371a17\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " Sep 17 18:29:55.590309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.590154 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-maas-api-tls\") pod \"5a8e356e-3830-4d95-a1c3-1aa456371a17\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " Sep 17 18:29:55.590309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.590183 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw5kr\" (UniqueName: \"kubernetes.io/projected/5a8e356e-3830-4d95-a1c3-1aa456371a17-kube-api-access-bw5kr\") pod \"5a8e356e-3830-4d95-a1c3-1aa456371a17\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " Sep 17 18:29:55.590484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.590320 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8e356e-3830-4d95-a1c3-1aa456371a17-openshift-service-ca\") pod \"5a8e356e-3830-4d95-a1c3-1aa456371a17\" (UID: \"5a8e356e-3830-4d95-a1c3-1aa456371a17\") " Sep 17 18:29:55.590707 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.590663 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8e356e-3830-4d95-a1c3-1aa456371a17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a8e356e-3830-4d95-a1c3-1aa456371a17" (UID: "5a8e356e-3830-4d95-a1c3-1aa456371a17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:29:55.590846 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.590785 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8e356e-3830-4d95-a1c3-1aa456371a17-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:55.592352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.592324 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8e356e-3830-4d95-a1c3-1aa456371a17-kube-api-access-bw5kr" (OuterVolumeSpecName: "kube-api-access-bw5kr") pod "5a8e356e-3830-4d95-a1c3-1aa456371a17" (UID: "5a8e356e-3830-4d95-a1c3-1aa456371a17"). InnerVolumeSpecName "kube-api-access-bw5kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:29:55.592471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.592453 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5a8e356e-3830-4d95-a1c3-1aa456371a17" (UID: "5a8e356e-3830-4d95-a1c3-1aa456371a17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:29:55.592673 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.592656 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a8e356e-3830-4d95-a1c3-1aa456371a17" (UID: "5a8e356e-3830-4d95-a1c3-1aa456371a17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:29:55.691794 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.691763 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:55.691794 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.691790 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a8e356e-3830-4d95-a1c3-1aa456371a17-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:55.691794 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.691799 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw5kr\" (UniqueName: \"kubernetes.io/projected/5a8e356e-3830-4d95-a1c3-1aa456371a17-kube-api-access-bw5kr\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:55.850351 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.850319 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn"] Sep 17 18:29:55.852590 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:55.852567 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4faba0-6cbfc87c98-gr4pn"] Sep 17 18:29:57.347577 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:57.347544 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8e356e-3830-4d95-a1c3-1aa456371a17" path="/var/lib/kubelet/pods/5a8e356e-3830-4d95-a1c3-1aa456371a17/volumes" Sep 17 18:29:57.978784 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:57.978743 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" podUID="4a507a93-f7b6-4d23-944d-70c47230218a" containerName="main" containerID="cri-o://280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4" gracePeriod=45 Sep 17 18:29:57.989469 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:57.989430 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" podUID="c6a795bf-6f73-494e-8bb8-bd9e23407bf5" containerName="main" containerID="cri-o://b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab" gracePeriod=45 Sep 17 18:29:58.243552 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.243489 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:29:58.283195 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.283170 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:29:58.318992 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.318959 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-dshm\") pod \"4a507a93-f7b6-4d23-944d-70c47230218a\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " Sep 17 18:29:58.319178 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.319018 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4z8b\" (UniqueName: \"kubernetes.io/projected/4a507a93-f7b6-4d23-944d-70c47230218a-kube-api-access-b4z8b\") pod \"4a507a93-f7b6-4d23-944d-70c47230218a\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " Sep 17 18:29:58.319178 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.319128 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-model-cache\") pod \"4a507a93-f7b6-4d23-944d-70c47230218a\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " Sep 17 18:29:58.319314 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.319184 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a507a93-f7b6-4d23-944d-70c47230218a-tls-certs\") pod \"4a507a93-f7b6-4d23-944d-70c47230218a\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " Sep 17 18:29:58.319314 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.319240 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-home\") pod \"4a507a93-f7b6-4d23-944d-70c47230218a\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " Sep 17 18:29:58.319314 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.319262 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-tmp-dir\") pod \"4a507a93-f7b6-4d23-944d-70c47230218a\" (UID: \"4a507a93-f7b6-4d23-944d-70c47230218a\") " Sep 17 18:29:58.319966 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.319775 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-home" (OuterVolumeSpecName: "home") pod "4a507a93-f7b6-4d23-944d-70c47230218a" (UID: "4a507a93-f7b6-4d23-944d-70c47230218a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:29:58.319966 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.319852 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a507a93-f7b6-4d23-944d-70c47230218a" (UID: "4a507a93-f7b6-4d23-944d-70c47230218a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:29:58.319966 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.319899 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a507a93-f7b6-4d23-944d-70c47230218a" (UID: "4a507a93-f7b6-4d23-944d-70c47230218a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:29:58.321733 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.321703 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a507a93-f7b6-4d23-944d-70c47230218a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a507a93-f7b6-4d23-944d-70c47230218a" (UID: "4a507a93-f7b6-4d23-944d-70c47230218a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:29:58.321733 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.321709 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-dshm" (OuterVolumeSpecName: "dshm") pod "4a507a93-f7b6-4d23-944d-70c47230218a" (UID: "4a507a93-f7b6-4d23-944d-70c47230218a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:29:58.322048 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.322025 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a507a93-f7b6-4d23-944d-70c47230218a-kube-api-access-b4z8b" (OuterVolumeSpecName: "kube-api-access-b4z8b") pod "4a507a93-f7b6-4d23-944d-70c47230218a" (UID: "4a507a93-f7b6-4d23-944d-70c47230218a"). InnerVolumeSpecName "kube-api-access-b4z8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:29:58.420801 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.420758 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-655p2\" (UniqueName: \"kubernetes.io/projected/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-kube-api-access-655p2\") pod \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " Sep 17 18:29:58.421295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.420826 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-dshm\") pod \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " Sep 17 18:29:58.421295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.420891 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tls-certs\") pod \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " Sep 17 18:29:58.421295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.420925 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-home\") pod \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " Sep 17 18:29:58.421295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.420949 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-model-cache\") pod \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " Sep 17 18:29:58.421295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421012 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tmp-dir\") pod \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\" (UID: \"c6a795bf-6f73-494e-8bb8-bd9e23407bf5\") " Sep 17 18:29:58.421295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421209 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-home" (OuterVolumeSpecName: "home") pod "c6a795bf-6f73-494e-8bb8-bd9e23407bf5" (UID: "c6a795bf-6f73-494e-8bb8-bd9e23407bf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:29:58.421295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421279 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6a795bf-6f73-494e-8bb8-bd9e23407bf5" (UID: "c6a795bf-6f73-494e-8bb8-bd9e23407bf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421350 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6a795bf-6f73-494e-8bb8-bd9e23407bf5" (UID: "c6a795bf-6f73-494e-8bb8-bd9e23407bf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421464 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421487 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4z8b\" (UniqueName: \"kubernetes.io/projected/4a507a93-f7b6-4d23-944d-70c47230218a-kube-api-access-b4z8b\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421504 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421518 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a507a93-f7b6-4d23-944d-70c47230218a-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421531 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421543 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421554 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421566 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a507a93-f7b6-4d23-944d-70c47230218a-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.421680 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.421578 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.422986 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.422967 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-dshm" (OuterVolumeSpecName: "dshm") pod "c6a795bf-6f73-494e-8bb8-bd9e23407bf5" (UID: "c6a795bf-6f73-494e-8bb8-bd9e23407bf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:29:58.423094 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.423073 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-kube-api-access-655p2" (OuterVolumeSpecName: "kube-api-access-655p2") pod "c6a795bf-6f73-494e-8bb8-bd9e23407bf5" (UID: "c6a795bf-6f73-494e-8bb8-bd9e23407bf5"). InnerVolumeSpecName "kube-api-access-655p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:29:58.423139 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.423108 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6a795bf-6f73-494e-8bb8-bd9e23407bf5" (UID: "c6a795bf-6f73-494e-8bb8-bd9e23407bf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:29:58.522716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.522640 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655p2\" (UniqueName: \"kubernetes.io/projected/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-kube-api-access-655p2\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.522716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.522665 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.522716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.522675 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a795bf-6f73-494e-8bb8-bd9e23407bf5-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:29:58.552553 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.552518 2582 generic.go:356] "Generic (PLEG): container finished" podID="4a507a93-f7b6-4d23-944d-70c47230218a" containerID="280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4" exitCode=0 Sep 17 18:29:58.552723 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.552557 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" Sep 17 18:29:58.552723 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.552597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" event={"ID":"4a507a93-f7b6-4d23-944d-70c47230218a","Type":"ContainerDied","Data":"280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4"} Sep 17 18:29:58.552723 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.552637 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb" event={"ID":"4a507a93-f7b6-4d23-944d-70c47230218a","Type":"ContainerDied","Data":"6331facf596dd5533704fc885bebb510fe94686a86d25ffb2a352e3552fe8384"} Sep 17 18:29:58.552723 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.552659 2582 scope.go:117] "RemoveContainer" containerID="280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4" Sep 17 18:29:58.554272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.554252 2582 generic.go:356] "Generic (PLEG): container finished" podID="c6a795bf-6f73-494e-8bb8-bd9e23407bf5" containerID="b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab" exitCode=0 Sep 17 18:29:58.554383 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.554293 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" Sep 17 18:29:58.554383 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.554315 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" event={"ID":"c6a795bf-6f73-494e-8bb8-bd9e23407bf5","Type":"ContainerDied","Data":"b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab"} Sep 17 18:29:58.554383 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.554353 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p" event={"ID":"c6a795bf-6f73-494e-8bb8-bd9e23407bf5","Type":"ContainerDied","Data":"d2873cab3a76facbf786e8cd188fdf633010b14f13ea046d87edd9156ace1c3b"} Sep 17 18:29:58.563676 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.563652 2582 scope.go:117] "RemoveContainer" containerID="280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4" Sep 17 18:29:58.564973 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:29:58.564048 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4\": container with ID starting with 280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4 not found: ID does not exist" containerID="280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4" Sep 17 18:29:58.564973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.564087 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4"} err="failed to get container status \"280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4\": rpc error: code = NotFound desc = could not find container \"280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4\": container with ID starting with 280fd29d2e83632e97e5806ebcc82d9d7ca95ed08182c5653cfb0b73948b60c4 not found: ID does not exist" Sep 17 18:29:58.564973 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.564112 2582 scope.go:117] "RemoveContainer" containerID="b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab" Sep 17 18:29:58.579005 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.578981 2582 scope.go:117] "RemoveContainer" containerID="b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab" Sep 17 18:29:58.579435 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:29:58.579414 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab\": container with ID starting with b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab not found: ID does not exist" containerID="b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab" Sep 17 18:29:58.579507 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.579442 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab"} err="failed to get container status \"b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab\": rpc error: code = NotFound desc = could not find container \"b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab\": container with ID starting with b9c42fe1e8e2d9b995c8733ca5bd78bac20c25bf9fa3f61b4727a5abe0c9d9ab not found: ID does not exist" Sep 17 18:29:58.583652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.583626 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb"] Sep 17 18:29:58.586876 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.586836 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-w4-b18433-kserve-5768fc7b56-qvtcb"] Sep 17 18:29:58.597428 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.597403 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p"] Sep 17 18:29:58.601315 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:58.601292 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/premium-w4-b18433-kserve-69fdf5bc8f-tjr2p"] Sep 17 18:29:59.349840 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:59.349763 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a507a93-f7b6-4d23-944d-70c47230218a" path="/var/lib/kubelet/pods/4a507a93-f7b6-4d23-944d-70c47230218a/volumes" Sep 17 18:29:59.350256 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:29:59.350242 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a795bf-6f73-494e-8bb8-bd9e23407bf5" path="/var/lib/kubelet/pods/c6a795bf-6f73-494e-8bb8-bd9e23407bf5/volumes" Sep 17 18:30:18.309571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:18.309536 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr"] Sep 17 18:30:20.270087 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.270042 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v"] Sep 17 18:30:20.270555 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.270320 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" podUID="9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" containerName="maas-api" containerID="cri-o://36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c" gracePeriod=30 Sep 17 18:30:20.544283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.544248 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:30:20.658251 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.658222 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-metrics-certs\") pod \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " Sep 17 18:30:20.658251 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.658261 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtbtp\" (UniqueName: \"kubernetes.io/projected/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-kube-api-access-mtbtp\") pod \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " Sep 17 18:30:20.658463 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.658324 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-openshift-service-ca\") pod \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " Sep 17 18:30:20.658463 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.658409 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-maas-api-tls\") pod \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\" (UID: \"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2\") " Sep 17 18:30:20.658784 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.658760 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" (UID: "9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:20.660453 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.660428 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-kube-api-access-mtbtp" (OuterVolumeSpecName: "kube-api-access-mtbtp") pod "9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" (UID: "9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2"). InnerVolumeSpecName "kube-api-access-mtbtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:20.660559 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.660456 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" (UID: "9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:20.660613 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.660561 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" (UID: "9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:20.695550 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.695518 2582 generic.go:356] "Generic (PLEG): container finished" podID="9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" containerID="36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c" exitCode=0 Sep 17 18:30:20.695691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.695570 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" Sep 17 18:30:20.695691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.695597 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" event={"ID":"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2","Type":"ContainerDied","Data":"36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c"} Sep 17 18:30:20.695691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.695664 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v" event={"ID":"9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2","Type":"ContainerDied","Data":"8ca7969b8eb3353c5e700fde28378bc9b0980939ad78612ad9c6aa3d0578e99b"} Sep 17 18:30:20.695931 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.695693 2582 scope.go:117] "RemoveContainer" containerID="36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c" Sep 17 18:30:20.705990 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.705969 2582 scope.go:117] "RemoveContainer" containerID="36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c" Sep 17 18:30:20.706247 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:30:20.706231 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c\": container with ID starting with 36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c not found: ID does not exist" containerID="36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c" Sep 17 18:30:20.706303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.706256 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c"} err="failed to get container status \"36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c\": rpc error: code = NotFound desc = could not find container \"36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c\": container with ID starting with 36a38fa9785c1fbf40a3cd82fec4a7482e00d2dc211b4d67d6bd1441f9d2d98c not found: ID does not exist" Sep 17 18:30:20.719030 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.719007 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v"] Sep 17 18:30:20.723616 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.723593 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b06908-7985d99b86-rg57v"] Sep 17 18:30:20.760014 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.759977 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtbtp\" (UniqueName: \"kubernetes.io/projected/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-kube-api-access-mtbtp\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:20.760014 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.760015 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:20.760283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.760032 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:20.760283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:20.760045 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:21.347522 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:21.347482 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" path="/var/lib/kubelet/pods/9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2/volumes" Sep 17 18:30:31.993997 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:31.993960 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b"] Sep 17 18:30:31.994369 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:31.994295 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" podUID="ce958664-dfe0-454a-aae9-bd23d9844de0" containerName="maas-api" containerID="cri-o://9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9" gracePeriod=30 Sep 17 18:30:32.258009 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.257937 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:30:32.380131 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.380100 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-metrics-certs\") pod \"ce958664-dfe0-454a-aae9-bd23d9844de0\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " Sep 17 18:30:32.380320 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.380153 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce958664-dfe0-454a-aae9-bd23d9844de0-openshift-service-ca\") pod \"ce958664-dfe0-454a-aae9-bd23d9844de0\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " Sep 17 18:30:32.380320 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.380206 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls\") pod \"ce958664-dfe0-454a-aae9-bd23d9844de0\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " Sep 17 18:30:32.380320 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.380251 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgttd\" (UniqueName: \"kubernetes.io/projected/ce958664-dfe0-454a-aae9-bd23d9844de0-kube-api-access-kgttd\") pod \"ce958664-dfe0-454a-aae9-bd23d9844de0\" (UID: \"ce958664-dfe0-454a-aae9-bd23d9844de0\") " Sep 17 18:30:32.380558 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.380531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce958664-dfe0-454a-aae9-bd23d9844de0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ce958664-dfe0-454a-aae9-bd23d9844de0" (UID: "ce958664-dfe0-454a-aae9-bd23d9844de0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:32.382309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.382277 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce958664-dfe0-454a-aae9-bd23d9844de0-kube-api-access-kgttd" (OuterVolumeSpecName: "kube-api-access-kgttd") pod "ce958664-dfe0-454a-aae9-bd23d9844de0" (UID: "ce958664-dfe0-454a-aae9-bd23d9844de0"). InnerVolumeSpecName "kube-api-access-kgttd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:32.382309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.382292 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ce958664-dfe0-454a-aae9-bd23d9844de0" (UID: "ce958664-dfe0-454a-aae9-bd23d9844de0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:32.382452 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.382325 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ce958664-dfe0-454a-aae9-bd23d9844de0" (UID: "ce958664-dfe0-454a-aae9-bd23d9844de0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:32.481857 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.481818 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgttd\" (UniqueName: \"kubernetes.io/projected/ce958664-dfe0-454a-aae9-bd23d9844de0-kube-api-access-kgttd\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.481857 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.481847 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.481857 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.481873 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce958664-dfe0-454a-aae9-bd23d9844de0-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.481857 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.481886 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce958664-dfe0-454a-aae9-bd23d9844de0-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:32.765551 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.765514 2582 generic.go:356] "Generic (PLEG): container finished" podID="ce958664-dfe0-454a-aae9-bd23d9844de0" containerID="9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9" exitCode=0 Sep 17 18:30:32.765737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.765565 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" Sep 17 18:30:32.765737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.765603 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" event={"ID":"ce958664-dfe0-454a-aae9-bd23d9844de0","Type":"ContainerDied","Data":"9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9"} Sep 17 18:30:32.765737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.765665 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b" event={"ID":"ce958664-dfe0-454a-aae9-bd23d9844de0","Type":"ContainerDied","Data":"3d9d5b3b08349ffe8bc6e761bfc233a72ac8ce13824d18e34708e5b8160c7240"} Sep 17 18:30:32.765737 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.765692 2582 scope.go:117] "RemoveContainer" containerID="9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9" Sep 17 18:30:32.775808 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.775791 2582 scope.go:117] "RemoveContainer" containerID="9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9" Sep 17 18:30:32.776110 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:30:32.776090 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9\": container with ID starting with 9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9 not found: ID does not exist" containerID="9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9" Sep 17 18:30:32.776177 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.776120 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9"} err="failed to get container status \"9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9\": rpc error: code = NotFound desc = could not find container \"9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9\": container with ID starting with 9042d0b30607129caa6d81384382b641e7eff4877770c5d6264f107a4ed4aaf9 not found: ID does not exist" Sep 17 18:30:32.797365 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.797339 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b"] Sep 17 18:30:32.808011 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:32.807984 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-5d8053-75765dcfdf-lmx6b"] Sep 17 18:30:33.330098 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.330061 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" podUID="5784409b-49f0-4286-80eb-baae97b50e91" containerName="main" containerID="cri-o://6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897" gracePeriod=45 Sep 17 18:30:33.349133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.349108 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce958664-dfe0-454a-aae9-bd23d9844de0" path="/var/lib/kubelet/pods/ce958664-dfe0-454a-aae9-bd23d9844de0/volumes" Sep 17 18:30:33.592241 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.592179 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:30:33.693160 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693130 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5784409b-49f0-4286-80eb-baae97b50e91-tls-certs\") pod \"5784409b-49f0-4286-80eb-baae97b50e91\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " Sep 17 18:30:33.693357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693168 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-home\") pod \"5784409b-49f0-4286-80eb-baae97b50e91\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " Sep 17 18:30:33.693357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693268 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcrpn\" (UniqueName: \"kubernetes.io/projected/5784409b-49f0-4286-80eb-baae97b50e91-kube-api-access-tcrpn\") pod \"5784409b-49f0-4286-80eb-baae97b50e91\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " Sep 17 18:30:33.693357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693318 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-tmp-dir\") pod \"5784409b-49f0-4286-80eb-baae97b50e91\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " Sep 17 18:30:33.693535 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693358 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-model-cache\") pod \"5784409b-49f0-4286-80eb-baae97b50e91\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " Sep 17 18:30:33.693535 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693394 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-dshm\") pod \"5784409b-49f0-4286-80eb-baae97b50e91\" (UID: \"5784409b-49f0-4286-80eb-baae97b50e91\") " Sep 17 18:30:33.693535 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693471 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-home" (OuterVolumeSpecName: "home") pod "5784409b-49f0-4286-80eb-baae97b50e91" (UID: "5784409b-49f0-4286-80eb-baae97b50e91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:33.693706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693618 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5784409b-49f0-4286-80eb-baae97b50e91" (UID: "5784409b-49f0-4286-80eb-baae97b50e91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:33.693792 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693775 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.693852 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693797 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.693852 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.693830 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-model-cache" (OuterVolumeSpecName: "model-cache") pod "5784409b-49f0-4286-80eb-baae97b50e91" (UID: "5784409b-49f0-4286-80eb-baae97b50e91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:33.695790 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.695765 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-dshm" (OuterVolumeSpecName: "dshm") pod "5784409b-49f0-4286-80eb-baae97b50e91" (UID: "5784409b-49f0-4286-80eb-baae97b50e91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:30:33.695956 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.695936 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5784409b-49f0-4286-80eb-baae97b50e91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5784409b-49f0-4286-80eb-baae97b50e91" (UID: "5784409b-49f0-4286-80eb-baae97b50e91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:33.696040 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.695951 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5784409b-49f0-4286-80eb-baae97b50e91-kube-api-access-tcrpn" (OuterVolumeSpecName: "kube-api-access-tcrpn") pod "5784409b-49f0-4286-80eb-baae97b50e91" (UID: "5784409b-49f0-4286-80eb-baae97b50e91"). InnerVolumeSpecName "kube-api-access-tcrpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:33.773060 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.773026 2582 generic.go:356] "Generic (PLEG): container finished" podID="5784409b-49f0-4286-80eb-baae97b50e91" containerID="6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897" exitCode=0 Sep 17 18:30:33.773219 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.773070 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" Sep 17 18:30:33.773219 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.773110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" event={"ID":"5784409b-49f0-4286-80eb-baae97b50e91","Type":"ContainerDied","Data":"6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897"} Sep 17 18:30:33.773219 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.773164 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr" event={"ID":"5784409b-49f0-4286-80eb-baae97b50e91","Type":"ContainerDied","Data":"a510b7badc173ba2eafc8f792b55dd13b792847ef3da9755cca7d54d3f13e094"} Sep 17 18:30:33.773219 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.773187 2582 scope.go:117] "RemoveContainer" containerID="6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897" Sep 17 18:30:33.783268 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.783248 2582 scope.go:117] "RemoveContainer" containerID="6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897" Sep 17 18:30:33.783545 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:30:33.783525 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897\": container with ID starting with 6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897 not found: ID does not exist" containerID="6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897" Sep 17 18:30:33.783597 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.783553 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897"} err="failed to get container status \"6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897\": rpc error: code = NotFound desc = could not find container \"6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897\": container with ID starting with 6b72f48332ff0bcbb705a3cc5f9712005150b4650148a7e60a68ee98448b2897 not found: ID does not exist" Sep 17 18:30:33.794464 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.794435 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcrpn\" (UniqueName: \"kubernetes.io/projected/5784409b-49f0-4286-80eb-baae97b50e91-kube-api-access-tcrpn\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.794577 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.794476 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.794577 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.794487 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5784409b-49f0-4286-80eb-baae97b50e91-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.794577 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.794495 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5784409b-49f0-4286-80eb-baae97b50e91-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:33.795693 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.795675 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr"] Sep 17 18:30:33.801526 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:33.801504 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/distinct-2-w0-5d8053-kserve-857bd8c7f7-mblrr"] Sep 17 18:30:35.348100 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:35.348067 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5784409b-49f0-4286-80eb-baae97b50e91" path="/var/lib/kubelet/pods/5784409b-49f0-4286-80eb-baae97b50e91/volumes" Sep 17 18:30:38.734349 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:38.734312 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx"] Sep 17 18:30:38.734841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:38.734554 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" podUID="c540d39a-9710-4b13-b3f8-2d53cf955701" containerName="maas-api" containerID="cri-o://bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166" gracePeriod=30 Sep 17 18:30:38.990700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:38.990641 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:30:39.151317 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.151283 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-metrics-certs\") pod \"c540d39a-9710-4b13-b3f8-2d53cf955701\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " Sep 17 18:30:39.151500 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.151343 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c540d39a-9710-4b13-b3f8-2d53cf955701-openshift-service-ca\") pod \"c540d39a-9710-4b13-b3f8-2d53cf955701\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " Sep 17 18:30:39.151500 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.151378 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqwqn\" (UniqueName: \"kubernetes.io/projected/c540d39a-9710-4b13-b3f8-2d53cf955701-kube-api-access-xqwqn\") pod \"c540d39a-9710-4b13-b3f8-2d53cf955701\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " Sep 17 18:30:39.151500 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.151437 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-maas-api-tls\") pod \"c540d39a-9710-4b13-b3f8-2d53cf955701\" (UID: \"c540d39a-9710-4b13-b3f8-2d53cf955701\") " Sep 17 18:30:39.151897 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.151849 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c540d39a-9710-4b13-b3f8-2d53cf955701-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c540d39a-9710-4b13-b3f8-2d53cf955701" (UID: "c540d39a-9710-4b13-b3f8-2d53cf955701"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:39.153564 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.153535 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c540d39a-9710-4b13-b3f8-2d53cf955701" (UID: "c540d39a-9710-4b13-b3f8-2d53cf955701"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:39.153663 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.153627 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c540d39a-9710-4b13-b3f8-2d53cf955701" (UID: "c540d39a-9710-4b13-b3f8-2d53cf955701"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:39.154106 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.154078 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c540d39a-9710-4b13-b3f8-2d53cf955701-kube-api-access-xqwqn" (OuterVolumeSpecName: "kube-api-access-xqwqn") pod "c540d39a-9710-4b13-b3f8-2d53cf955701" (UID: "c540d39a-9710-4b13-b3f8-2d53cf955701"). InnerVolumeSpecName "kube-api-access-xqwqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:39.253209 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.253116 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:39.253209 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.253145 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c540d39a-9710-4b13-b3f8-2d53cf955701-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:39.253209 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.253155 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwqn\" (UniqueName: \"kubernetes.io/projected/c540d39a-9710-4b13-b3f8-2d53cf955701-kube-api-access-xqwqn\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:39.253209 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.253167 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c540d39a-9710-4b13-b3f8-2d53cf955701-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:39.815175 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.815142 2582 generic.go:356] "Generic (PLEG): container finished" podID="c540d39a-9710-4b13-b3f8-2d53cf955701" containerID="bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166" exitCode=0 Sep 17 18:30:39.815575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.815185 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" Sep 17 18:30:39.815575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.815231 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" event={"ID":"c540d39a-9710-4b13-b3f8-2d53cf955701","Type":"ContainerDied","Data":"bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166"} Sep 17 18:30:39.815575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.815283 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx" event={"ID":"c540d39a-9710-4b13-b3f8-2d53cf955701","Type":"ContainerDied","Data":"f32c6d0ab2b4813a9b2adc4f252dc62ce99a3ca572dc2b59f678d5b3faff9309"} Sep 17 18:30:39.815575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.815305 2582 scope.go:117] "RemoveContainer" containerID="bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166" Sep 17 18:30:39.824278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.824261 2582 scope.go:117] "RemoveContainer" containerID="bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166" Sep 17 18:30:39.824575 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:30:39.824559 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166\": container with ID starting with bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166 not found: ID does not exist" containerID="bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166" Sep 17 18:30:39.824649 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.824586 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166"} err="failed to get container status \"bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166\": rpc error: code = NotFound desc = could not find container \"bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166\": container with ID starting with bb41b19c5afae0ee50559fad98c8e08c6c9a1795b2804423b847b8ca76f3b166 not found: ID does not exist" Sep 17 18:30:39.866889 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.866779 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx"] Sep 17 18:30:39.880738 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:39.880696 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-1a9245-976c5b98f-f5lqx"] Sep 17 18:30:41.350272 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:41.350239 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c540d39a-9710-4b13-b3f8-2d53cf955701" path="/var/lib/kubelet/pods/c540d39a-9710-4b13-b3f8-2d53cf955701/volumes" Sep 17 18:30:57.017669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.017630 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv"] Sep 17 18:30:57.018202 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.017912 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" podUID="d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" containerName="maas-api" containerID="cri-o://573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f" gracePeriod=30 Sep 17 18:30:57.289103 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.289039 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:30:57.331586 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.331558 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvc89\" (UniqueName: \"kubernetes.io/projected/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-kube-api-access-lvc89\") pod \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " Sep 17 18:30:57.331757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.331642 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-maas-api-tls\") pod \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " Sep 17 18:30:57.331757 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.331686 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-metrics-certs\") pod \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " Sep 17 18:30:57.331856 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.331759 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-openshift-service-ca\") pod \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\" (UID: \"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec\") " Sep 17 18:30:57.332193 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.332144 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" (UID: "d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:30:57.334178 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.334143 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-kube-api-access-lvc89" (OuterVolumeSpecName: "kube-api-access-lvc89") pod "d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" (UID: "d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec"). InnerVolumeSpecName "kube-api-access-lvc89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:30:57.334393 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.334368 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" (UID: "d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:57.334477 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.334390 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" (UID: "d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:30:57.433034 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.432989 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:57.433034 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.433016 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvc89\" (UniqueName: \"kubernetes.io/projected/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-kube-api-access-lvc89\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:57.433034 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.433027 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:57.433034 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.433036 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:30:57.922990 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.922951 2582 generic.go:356] "Generic (PLEG): container finished" podID="d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" containerID="573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f" exitCode=0 Sep 17 18:30:57.923171 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.923004 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" Sep 17 18:30:57.923171 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.923040 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" event={"ID":"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec","Type":"ContainerDied","Data":"573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f"} Sep 17 18:30:57.923171 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.923099 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv" event={"ID":"d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec","Type":"ContainerDied","Data":"e09fe6150496d3ccd90c1f0ecbc48cfb4243bd8d263b49ebdcada7398eb60fb5"} Sep 17 18:30:57.923171 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.923126 2582 scope.go:117] "RemoveContainer" containerID="573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f" Sep 17 18:30:57.932956 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.932933 2582 scope.go:117] "RemoveContainer" containerID="573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f" Sep 17 18:30:57.933235 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:30:57.933218 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f\": container with ID starting with 573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f not found: ID does not exist" containerID="573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f" Sep 17 18:30:57.933283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.933244 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f"} err="failed to get container status \"573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f\": rpc error: code = NotFound desc = could not find container \"573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f\": container with ID starting with 573ca67c3d10b207d6a8743358088ed4d7ba4e1f263302a4d5a7cc707112f78f not found: ID does not exist" Sep 17 18:30:57.943660 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.943632 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv"] Sep 17 18:30:57.948060 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:57.948038 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-03270e-758df4bfd7-vvnbv"] Sep 17 18:30:59.346760 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:30:59.346730 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" path="/var/lib/kubelet/pods/d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec/volumes" Sep 17 18:31:28.862694 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.862618 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7631-5dfc16"] Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863082 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863095 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863107 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c540d39a-9710-4b13-b3f8-2d53cf955701" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863113 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c540d39a-9710-4b13-b3f8-2d53cf955701" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863121 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b42d62b-e740-4a13-8864-784ad5829e6a" containerName="payload-processing" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863127 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b42d62b-e740-4a13-8864-784ad5829e6a" containerName="payload-processing" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863135 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a795bf-6f73-494e-8bb8-bd9e23407bf5" containerName="main" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863140 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a795bf-6f73-494e-8bb8-bd9e23407bf5" containerName="main" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863146 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8e356e-3830-4d95-a1c3-1aa456371a17" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863151 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8e356e-3830-4d95-a1c3-1aa456371a17" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863159 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce958664-dfe0-454a-aae9-bd23d9844de0" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863163 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce958664-dfe0-454a-aae9-bd23d9844de0" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863170 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" containerName="maas-api" Sep 17 18:31:28.863174 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863175 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" containerName="maas-api" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863183 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5784409b-49f0-4286-80eb-baae97b50e91" containerName="main" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863188 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5784409b-49f0-4286-80eb-baae97b50e91" containerName="main" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863199 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56258a1-d9fa-43ee-8495-88e71277f744" containerName="maas-api" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863203 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56258a1-d9fa-43ee-8495-88e71277f744" containerName="maas-api" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863208 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a507a93-f7b6-4d23-944d-70c47230218a" containerName="main" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863214 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a507a93-f7b6-4d23-944d-70c47230218a" containerName="main" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863268 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d868cde7-cbc3-4a46-b7e0-f45d2dcd5bec" containerName="maas-api" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863277 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c540d39a-9710-4b13-b3f8-2d53cf955701" containerName="maas-api" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863282 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5784409b-49f0-4286-80eb-baae97b50e91" containerName="main" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863287 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ec9d9b8-cb8d-4d7b-b615-aa50618db1f2" containerName="maas-api" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863293 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8e356e-3830-4d95-a1c3-1aa456371a17" containerName="maas-api" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863298 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56258a1-d9fa-43ee-8495-88e71277f744" containerName="maas-api" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863303 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b42d62b-e740-4a13-8864-784ad5829e6a" containerName="payload-processing" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863309 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a795bf-6f73-494e-8bb8-bd9e23407bf5" containerName="main" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863314 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a507a93-f7b6-4d23-944d-70c47230218a" containerName="main" Sep 17 18:31:28.863614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.863319 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce958664-dfe0-454a-aae9-bd23d9844de0" containerName="maas-api" Sep 17 18:31:28.868217 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.868202 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-5dfc16" Sep 17 18:31:28.877781 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.877765 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5zgf\"" Sep 17 18:31:28.891629 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.891603 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7631-5dfc16"] Sep 17 18:31:28.942768 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:28.942736 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tbh4\" (UniqueName: \"kubernetes.io/projected/55da9dcb-592a-46fa-8a9e-0a9a469901f8-kube-api-access-7tbh4\") pod \"test-curl-7631-5dfc16\" (UID: \"55da9dcb-592a-46fa-8a9e-0a9a469901f8\") " pod="opendatahub/test-curl-7631-5dfc16" Sep 17 18:31:29.044061 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:29.044016 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tbh4\" (UniqueName: \"kubernetes.io/projected/55da9dcb-592a-46fa-8a9e-0a9a469901f8-kube-api-access-7tbh4\") pod \"test-curl-7631-5dfc16\" (UID: \"55da9dcb-592a-46fa-8a9e-0a9a469901f8\") " pod="opendatahub/test-curl-7631-5dfc16" Sep 17 18:31:29.052328 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:29.052291 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tbh4\" (UniqueName: \"kubernetes.io/projected/55da9dcb-592a-46fa-8a9e-0a9a469901f8-kube-api-access-7tbh4\") pod \"test-curl-7631-5dfc16\" (UID: \"55da9dcb-592a-46fa-8a9e-0a9a469901f8\") " pod="opendatahub/test-curl-7631-5dfc16" Sep 17 18:31:29.177582 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:29.177546 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-5dfc16" Sep 17 18:31:29.449133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:29.449104 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7631-5dfc16"] Sep 17 18:31:29.451377 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:31:29.451350 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55da9dcb_592a_46fa_8a9e_0a9a469901f8.slice/crio-a6a71d51e588474a5e02db5e3f02daf96eb2177946c01aa133476c5968393055 WatchSource:0}: Error finding container a6a71d51e588474a5e02db5e3f02daf96eb2177946c01aa133476c5968393055: Status 404 returned error can't find the container with id a6a71d51e588474a5e02db5e3f02daf96eb2177946c01aa133476c5968393055 Sep 17 18:31:30.110792 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:30.110752 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7631-5dfc16" event={"ID":"55da9dcb-592a-46fa-8a9e-0a9a469901f8","Type":"ContainerStarted","Data":"c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62"} Sep 17 18:31:30.111309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:30.110801 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7631-5dfc16" event={"ID":"55da9dcb-592a-46fa-8a9e-0a9a469901f8","Type":"ContainerStarted","Data":"a6a71d51e588474a5e02db5e3f02daf96eb2177946c01aa133476c5968393055"} Sep 17 18:31:30.126550 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:30.126503 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7631-5dfc16" podStartSLOduration=2.126486947 podStartE2EDuration="2.126486947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:30.125152581 +0000 UTC m=+1701.189096539" watchObservedRunningTime="2026-09-17 18:31:30.126486947 +0000 UTC m=+1701.190430969" Sep 17 18:31:30.795467 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:30.795384 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7631-5dfc16"] Sep 17 18:31:30.799314 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:30.799284 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7631-5dfc16"] Sep 17 18:31:31.116152 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:31:31.116072 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7631-5dfc16" container="test-curl-7631-5dfc16" Sep 17 18:31:32.113815 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:32.113782 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph"] Sep 17 18:31:32.122520 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:32.122469 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7631-5dfc16" podUID="55da9dcb-592a-46fa-8a9e-0a9a469901f8" containerName="test-curl-7631-5dfc16" containerID="cri-o://c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62" gracePeriod=2 Sep 17 18:31:34.309372 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:34.309351 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-5dfc16" Sep 17 18:31:34.396995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:34.396914 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tbh4\" (UniqueName: \"kubernetes.io/projected/55da9dcb-592a-46fa-8a9e-0a9a469901f8-kube-api-access-7tbh4\") pod \"55da9dcb-592a-46fa-8a9e-0a9a469901f8\" (UID: \"55da9dcb-592a-46fa-8a9e-0a9a469901f8\") " Sep 17 18:31:34.398964 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:34.398940 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55da9dcb-592a-46fa-8a9e-0a9a469901f8-kube-api-access-7tbh4" (OuterVolumeSpecName: "kube-api-access-7tbh4") pod "55da9dcb-592a-46fa-8a9e-0a9a469901f8" (UID: "55da9dcb-592a-46fa-8a9e-0a9a469901f8"). InnerVolumeSpecName "kube-api-access-7tbh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:34.497802 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:34.497773 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tbh4\" (UniqueName: \"kubernetes.io/projected/55da9dcb-592a-46fa-8a9e-0a9a469901f8-kube-api-access-7tbh4\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:31:35.140471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:35.140434 2582 generic.go:356] "Generic (PLEG): container finished" podID="55da9dcb-592a-46fa-8a9e-0a9a469901f8" containerID="c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62" exitCode=137 Sep 17 18:31:35.140669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:35.140487 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-5dfc16" Sep 17 18:31:35.140669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:35.140494 2582 scope.go:117] "RemoveContainer" containerID="c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62" Sep 17 18:31:35.150085 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:35.150065 2582 scope.go:117] "RemoveContainer" containerID="c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62" Sep 17 18:31:35.150417 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:31:35.150396 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62\": container with ID starting with c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62 not found: ID does not exist" containerID="c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62" Sep 17 18:31:35.150519 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:35.150425 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62"} err="failed to get container status \"c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62\": rpc error: code = NotFound desc = could not find container \"c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62\": container with ID starting with c183bafb4971bbb927e143e44b5fbab0897e8655cf9624f00396dcd115aebb62 not found: ID does not exist" Sep 17 18:31:35.153520 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:31:35.153487 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7631-5dfc16\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="55da9dcb-592a-46fa-8a9e-0a9a469901f8" pod="opendatahub/test-curl-7631-5dfc16" Sep 17 18:31:35.346031 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:35.345999 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55da9dcb-592a-46fa-8a9e-0a9a469901f8" path="/var/lib/kubelet/pods/55da9dcb-592a-46fa-8a9e-0a9a469901f8/volumes" Sep 17 18:31:42.909716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.909681 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9"] Sep 17 18:31:42.910281 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.910260 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55da9dcb-592a-46fa-8a9e-0a9a469901f8" containerName="test-curl-7631-5dfc16" Sep 17 18:31:42.910281 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.910283 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="55da9dcb-592a-46fa-8a9e-0a9a469901f8" containerName="test-curl-7631-5dfc16" Sep 17 18:31:42.910544 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.910458 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="55da9dcb-592a-46fa-8a9e-0a9a469901f8" containerName="test-curl-7631-5dfc16" Sep 17 18:31:42.913682 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.913667 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:42.916482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.916459 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"sub-test-model-95a92b-kserve-self-signed-certs\"" Sep 17 18:31:42.924822 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.924792 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9"] Sep 17 18:31:42.983633 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.983602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-home\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:42.983795 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.983638 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplx8\" (UniqueName: \"kubernetes.io/projected/f759e79a-daf6-456c-a8e8-796c627f5789-kube-api-access-mplx8\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:42.983795 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.983717 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-model-cache\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:42.983795 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.983747 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f759e79a-daf6-456c-a8e8-796c627f5789-tls-certs\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:42.983795 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.983779 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-tmp-dir\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:42.983958 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:42.983807 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-dshm\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.084847 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.084813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-home\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.084847 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.084849 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplx8\" (UniqueName: \"kubernetes.io/projected/f759e79a-daf6-456c-a8e8-796c627f5789-kube-api-access-mplx8\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.085133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.084958 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-model-cache\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.085133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.084995 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f759e79a-daf6-456c-a8e8-796c627f5789-tls-certs\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.085133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.085020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-tmp-dir\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.085133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.085054 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-dshm\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.085343 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.085279 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-home\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.085403 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.085373 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-model-cache\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.085461 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.085408 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-tmp-dir\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.087371 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.087343 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-dshm\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.087604 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.087587 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f759e79a-daf6-456c-a8e8-796c627f5789-tls-certs\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.093254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.093232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplx8\" (UniqueName: \"kubernetes.io/projected/f759e79a-daf6-456c-a8e8-796c627f5789-kube-api-access-mplx8\") pod \"sub-test-model-95a92b-kserve-5d87787dd7-424w9\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.223288 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.223203 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:43.500300 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:43.500274 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9"] Sep 17 18:31:43.502540 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:31:43.502512 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf759e79a_daf6_456c_a8e8_796c627f5789.slice/crio-6bfdd571045cb0cfe351fe3ec95670a2424c57b17ccd01303e4eaaeef74978c9 WatchSource:0}: Error finding container 6bfdd571045cb0cfe351fe3ec95670a2424c57b17ccd01303e4eaaeef74978c9: Status 404 returned error can't find the container with id 6bfdd571045cb0cfe351fe3ec95670a2424c57b17ccd01303e4eaaeef74978c9 Sep 17 18:31:44.193727 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:44.193687 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" event={"ID":"f759e79a-daf6-456c-a8e8-796c627f5789","Type":"ContainerStarted","Data":"97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741"} Sep 17 18:31:44.193727 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:44.193731 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" event={"ID":"f759e79a-daf6-456c-a8e8-796c627f5789","Type":"ContainerStarted","Data":"6bfdd571045cb0cfe351fe3ec95670a2424c57b17ccd01303e4eaaeef74978c9"} Sep 17 18:31:44.212706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:44.212633 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" podStartSLOduration=2.212618375 podStartE2EDuration="2.212618375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:44.212219908 +0000 UTC m=+1715.276163869" watchObservedRunningTime="2026-09-17 18:31:44.212618375 +0000 UTC m=+1715.276562332" Sep 17 18:31:47.134278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.134239 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" podUID="bf805383-407b-4dcc-850e-c1933724a004" containerName="main" containerID="cri-o://11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1" gracePeriod=45 Sep 17 18:31:47.399836 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.399814 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:31:47.528115 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.528083 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-dshm\") pod \"bf805383-407b-4dcc-850e-c1933724a004\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " Sep 17 18:31:47.528310 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.528136 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6q4n\" (UniqueName: \"kubernetes.io/projected/bf805383-407b-4dcc-850e-c1933724a004-kube-api-access-m6q4n\") pod \"bf805383-407b-4dcc-850e-c1933724a004\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " Sep 17 18:31:47.528310 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.528194 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf805383-407b-4dcc-850e-c1933724a004-tls-certs\") pod \"bf805383-407b-4dcc-850e-c1933724a004\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " Sep 17 18:31:47.528310 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.528257 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-model-cache\") pod \"bf805383-407b-4dcc-850e-c1933724a004\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " Sep 17 18:31:47.528501 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.528421 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-tmp-dir\") pod \"bf805383-407b-4dcc-850e-c1933724a004\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " Sep 17 18:31:47.528595 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.528576 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-home\") pod \"bf805383-407b-4dcc-850e-c1933724a004\" (UID: \"bf805383-407b-4dcc-850e-c1933724a004\") " Sep 17 18:31:47.529138 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.528798 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf805383-407b-4dcc-850e-c1933724a004" (UID: "bf805383-407b-4dcc-850e-c1933724a004"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:31:47.529138 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.529099 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:31:47.529378 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.529232 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-home" (OuterVolumeSpecName: "home") pod "bf805383-407b-4dcc-850e-c1933724a004" (UID: "bf805383-407b-4dcc-850e-c1933724a004"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:31:47.529562 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.529531 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf805383-407b-4dcc-850e-c1933724a004" (UID: "bf805383-407b-4dcc-850e-c1933724a004"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:31:47.530688 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.530658 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf805383-407b-4dcc-850e-c1933724a004-kube-api-access-m6q4n" (OuterVolumeSpecName: "kube-api-access-m6q4n") pod "bf805383-407b-4dcc-850e-c1933724a004" (UID: "bf805383-407b-4dcc-850e-c1933724a004"). InnerVolumeSpecName "kube-api-access-m6q4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:31:47.530786 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.530721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf805383-407b-4dcc-850e-c1933724a004-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf805383-407b-4dcc-850e-c1933724a004" (UID: "bf805383-407b-4dcc-850e-c1933724a004"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:31:47.530786 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.530721 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-dshm" (OuterVolumeSpecName: "dshm") pod "bf805383-407b-4dcc-850e-c1933724a004" (UID: "bf805383-407b-4dcc-850e-c1933724a004"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:31:47.630493 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.630453 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf805383-407b-4dcc-850e-c1933724a004-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:31:47.630493 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.630485 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:31:47.630493 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.630497 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:31:47.630734 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.630507 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf805383-407b-4dcc-850e-c1933724a004-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:31:47.630734 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:47.630520 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6q4n\" (UniqueName: \"kubernetes.io/projected/bf805383-407b-4dcc-850e-c1933724a004-kube-api-access-m6q4n\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:31:48.215615 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.215581 2582 generic.go:356] "Generic (PLEG): container finished" podID="bf805383-407b-4dcc-850e-c1933724a004" containerID="11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1" exitCode=0 Sep 17 18:31:48.216112 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.215624 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" Sep 17 18:31:48.216112 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.215672 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" event={"ID":"bf805383-407b-4dcc-850e-c1933724a004","Type":"ContainerDied","Data":"11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1"} Sep 17 18:31:48.216112 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.215721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph" event={"ID":"bf805383-407b-4dcc-850e-c1933724a004","Type":"ContainerDied","Data":"1c8cb6873f445b7151a12b341d29bc35e22482b90194ec8e856ec00884abf708"} Sep 17 18:31:48.216112 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.215748 2582 scope.go:117] "RemoveContainer" containerID="11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1" Sep 17 18:31:48.227273 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.227240 2582 scope.go:117] "RemoveContainer" containerID="11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1" Sep 17 18:31:48.228822 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:31:48.228254 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1\": container with ID starting with 11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1 not found: ID does not exist" containerID="11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1" Sep 17 18:31:48.228822 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.228291 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1"} err="failed to get container status \"11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1\": rpc error: code = NotFound desc = could not find container \"11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1\": container with ID starting with 11dcbc7541ab7f2670c0c0104e0f08913ebf2bb80978c9e5a2a8c192e8a068f1 not found: ID does not exist" Sep 17 18:31:48.243454 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.243431 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph"] Sep 17 18:31:48.248183 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:48.248158 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/auth-test-model-95a92b-kserve-77ffd7cb7f-vs5ph"] Sep 17 18:31:49.347257 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:49.347216 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf805383-407b-4dcc-850e-c1933724a004" path="/var/lib/kubelet/pods/bf805383-407b-4dcc-850e-c1933724a004/volumes" Sep 17 18:31:53.223847 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:53.223813 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:53.224355 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:53.223884 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:53.236473 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:53.236449 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:31:53.257611 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:31:53.257582 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:32:56.001366 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.001331 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7631-e156fd"] Sep 17 18:32:56.001820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.001806 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf805383-407b-4dcc-850e-c1933724a004" containerName="main" Sep 17 18:32:56.001877 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.001821 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf805383-407b-4dcc-850e-c1933724a004" containerName="main" Sep 17 18:32:56.001917 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.001909 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf805383-407b-4dcc-850e-c1933724a004" containerName="main" Sep 17 18:32:56.005228 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.005208 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-e156fd" Sep 17 18:32:56.007508 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.007484 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5zgf\"" Sep 17 18:32:56.011641 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.011620 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7631-e156fd"] Sep 17 18:32:56.088354 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.088328 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66dd\" (UniqueName: \"kubernetes.io/projected/618538bf-c029-44ec-82e3-c83e0de9a4f2-kube-api-access-c66dd\") pod \"test-curl-7631-e156fd\" (UID: \"618538bf-c029-44ec-82e3-c83e0de9a4f2\") " pod="opendatahub/test-curl-7631-e156fd" Sep 17 18:32:56.189278 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.189251 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66dd\" (UniqueName: \"kubernetes.io/projected/618538bf-c029-44ec-82e3-c83e0de9a4f2-kube-api-access-c66dd\") pod \"test-curl-7631-e156fd\" (UID: \"618538bf-c029-44ec-82e3-c83e0de9a4f2\") " pod="opendatahub/test-curl-7631-e156fd" Sep 17 18:32:56.197915 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.197891 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66dd\" (UniqueName: \"kubernetes.io/projected/618538bf-c029-44ec-82e3-c83e0de9a4f2-kube-api-access-c66dd\") pod \"test-curl-7631-e156fd\" (UID: \"618538bf-c029-44ec-82e3-c83e0de9a4f2\") " pod="opendatahub/test-curl-7631-e156fd" Sep 17 18:32:56.316915 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.316829 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-e156fd" Sep 17 18:32:56.588197 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:56.588166 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7631-e156fd"] Sep 17 18:32:56.590163 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:32:56.590135 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod618538bf_c029_44ec_82e3_c83e0de9a4f2.slice/crio-88bd3118a39b6462debe550d40c136accc1ea56abd96e03b26c7507572fc6ce8 WatchSource:0}: Error finding container 88bd3118a39b6462debe550d40c136accc1ea56abd96e03b26c7507572fc6ce8: Status 404 returned error can't find the container with id 88bd3118a39b6462debe550d40c136accc1ea56abd96e03b26c7507572fc6ce8 Sep 17 18:32:57.599357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:57.599323 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7631-e156fd" event={"ID":"618538bf-c029-44ec-82e3-c83e0de9a4f2","Type":"ContainerStarted","Data":"7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048"} Sep 17 18:32:57.599816 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:57.599366 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7631-e156fd" event={"ID":"618538bf-c029-44ec-82e3-c83e0de9a4f2","Type":"ContainerStarted","Data":"88bd3118a39b6462debe550d40c136accc1ea56abd96e03b26c7507572fc6ce8"} Sep 17 18:32:57.615035 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:57.614986 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7631-e156fd" podStartSLOduration=2.614970999 podStartE2EDuration="2.614970999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:57.613925461 +0000 UTC m=+1788.677869420" watchObservedRunningTime="2026-09-17 18:32:57.614970999 +0000 UTC m=+1788.678914957" Sep 17 18:32:58.269793 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:58.269759 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7631-e156fd"] Sep 17 18:32:58.279882 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:58.277380 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7631-e156fd"] Sep 17 18:32:58.606226 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:32:58.606194 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7631-e156fd" container="test-curl-7631-e156fd" Sep 17 18:32:59.610826 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:32:59.610790 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7631-e156fd" podUID="618538bf-c029-44ec-82e3-c83e0de9a4f2" containerName="test-curl-7631-e156fd" containerID="cri-o://7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048" gracePeriod=2 Sep 17 18:33:01.723258 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:01.723187 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9"] Sep 17 18:33:01.775646 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:01.775617 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-e156fd" Sep 17 18:33:01.945541 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:01.945508 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c66dd\" (UniqueName: \"kubernetes.io/projected/618538bf-c029-44ec-82e3-c83e0de9a4f2-kube-api-access-c66dd\") pod \"618538bf-c029-44ec-82e3-c83e0de9a4f2\" (UID: \"618538bf-c029-44ec-82e3-c83e0de9a4f2\") " Sep 17 18:33:01.947647 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:01.947623 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/618538bf-c029-44ec-82e3-c83e0de9a4f2-kube-api-access-c66dd" (OuterVolumeSpecName: "kube-api-access-c66dd") pod "618538bf-c029-44ec-82e3-c83e0de9a4f2" (UID: "618538bf-c029-44ec-82e3-c83e0de9a4f2"). InnerVolumeSpecName "kube-api-access-c66dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:33:02.047057 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:02.047026 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66dd\" (UniqueName: \"kubernetes.io/projected/618538bf-c029-44ec-82e3-c83e0de9a4f2-kube-api-access-c66dd\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:33:02.628291 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:02.628208 2582 generic.go:356] "Generic (PLEG): container finished" podID="618538bf-c029-44ec-82e3-c83e0de9a4f2" containerID="7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048" exitCode=137 Sep 17 18:33:02.628291 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:02.628250 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-e156fd" Sep 17 18:33:02.628496 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:02.628301 2582 scope.go:117] "RemoveContainer" containerID="7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048" Sep 17 18:33:02.637575 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:02.637546 2582 scope.go:117] "RemoveContainer" containerID="7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048" Sep 17 18:33:02.637851 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:33:02.637834 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048\": container with ID starting with 7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048 not found: ID does not exist" containerID="7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048" Sep 17 18:33:02.637953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:02.637886 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048"} err="failed to get container status \"7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048\": rpc error: code = NotFound desc = could not find container \"7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048\": container with ID starting with 7d7c982e23132e1a9b95927f81389bb5eb5ec1de1829b016e1a9c8b2d24d7048 not found: ID does not exist" Sep 17 18:33:02.638648 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:33:02.638622 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7631-e156fd\" is forbidden: User \"system:node:ip-10-0-137-104.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-104.ec2.internal' and this object" podUID="618538bf-c029-44ec-82e3-c83e0de9a4f2" pod="opendatahub/test-curl-7631-e156fd" Sep 17 18:33:03.346203 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:03.346166 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="618538bf-c029-44ec-82e3-c83e0de9a4f2" path="/var/lib/kubelet/pods/618538bf-c029-44ec-82e3-c83e0de9a4f2/volumes" Sep 17 18:33:11.409162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:11.409134 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:33:11.413210 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:11.412624 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:33:11.420934 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:11.420914 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:33:11.423806 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:11.423789 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:33:16.746687 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:16.746644 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" podUID="f759e79a-daf6-456c-a8e8-796c627f5789" containerName="main" containerID="cri-o://97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741" gracePeriod=45 Sep 17 18:33:17.003216 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.003162 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:33:17.090148 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090120 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-home\") pod \"f759e79a-daf6-456c-a8e8-796c627f5789\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " Sep 17 18:33:17.090324 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090225 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-dshm\") pod \"f759e79a-daf6-456c-a8e8-796c627f5789\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " Sep 17 18:33:17.090324 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090285 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f759e79a-daf6-456c-a8e8-796c627f5789-tls-certs\") pod \"f759e79a-daf6-456c-a8e8-796c627f5789\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " Sep 17 18:33:17.090443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090327 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-model-cache\") pod \"f759e79a-daf6-456c-a8e8-796c627f5789\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " Sep 17 18:33:17.090443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090349 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mplx8\" (UniqueName: \"kubernetes.io/projected/f759e79a-daf6-456c-a8e8-796c627f5789-kube-api-access-mplx8\") pod \"f759e79a-daf6-456c-a8e8-796c627f5789\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " Sep 17 18:33:17.090443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090355 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-home" (OuterVolumeSpecName: "home") pod "f759e79a-daf6-456c-a8e8-796c627f5789" (UID: "f759e79a-daf6-456c-a8e8-796c627f5789"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:33:17.090443 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090364 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-tmp-dir\") pod \"f759e79a-daf6-456c-a8e8-796c627f5789\" (UID: \"f759e79a-daf6-456c-a8e8-796c627f5789\") " Sep 17 18:33:17.090653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090605 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-model-cache" (OuterVolumeSpecName: "model-cache") pod "f759e79a-daf6-456c-a8e8-796c627f5789" (UID: "f759e79a-daf6-456c-a8e8-796c627f5789"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:33:17.090653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090618 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:33:17.090653 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.090645 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f759e79a-daf6-456c-a8e8-796c627f5789" (UID: "f759e79a-daf6-456c-a8e8-796c627f5789"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:33:17.092310 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.092288 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-dshm" (OuterVolumeSpecName: "dshm") pod "f759e79a-daf6-456c-a8e8-796c627f5789" (UID: "f759e79a-daf6-456c-a8e8-796c627f5789"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:33:17.092403 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.092367 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f759e79a-daf6-456c-a8e8-796c627f5789-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f759e79a-daf6-456c-a8e8-796c627f5789" (UID: "f759e79a-daf6-456c-a8e8-796c627f5789"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:33:17.092403 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.092383 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f759e79a-daf6-456c-a8e8-796c627f5789-kube-api-access-mplx8" (OuterVolumeSpecName: "kube-api-access-mplx8") pod "f759e79a-daf6-456c-a8e8-796c627f5789" (UID: "f759e79a-daf6-456c-a8e8-796c627f5789"). InnerVolumeSpecName "kube-api-access-mplx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:33:17.192014 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.191985 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:33:17.192014 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.192010 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mplx8\" (UniqueName: \"kubernetes.io/projected/f759e79a-daf6-456c-a8e8-796c627f5789-kube-api-access-mplx8\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:33:17.192014 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.192022 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:33:17.192245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.192031 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f759e79a-daf6-456c-a8e8-796c627f5789-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:33:17.192245 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.192039 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f759e79a-daf6-456c-a8e8-796c627f5789-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:33:17.716696 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.716650 2582 generic.go:356] "Generic (PLEG): container finished" podID="f759e79a-daf6-456c-a8e8-796c627f5789" containerID="97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741" exitCode=0 Sep 17 18:33:17.716696 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.716689 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" Sep 17 18:33:17.716952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.716782 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" event={"ID":"f759e79a-daf6-456c-a8e8-796c627f5789","Type":"ContainerDied","Data":"97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741"} Sep 17 18:33:17.716952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.716834 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9" event={"ID":"f759e79a-daf6-456c-a8e8-796c627f5789","Type":"ContainerDied","Data":"6bfdd571045cb0cfe351fe3ec95670a2424c57b17ccd01303e4eaaeef74978c9"} Sep 17 18:33:17.716952 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.716856 2582 scope.go:117] "RemoveContainer" containerID="97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741" Sep 17 18:33:17.726856 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.726831 2582 scope.go:117] "RemoveContainer" containerID="97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741" Sep 17 18:33:17.727219 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:33:17.727196 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741\": container with ID starting with 97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741 not found: ID does not exist" containerID="97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741" Sep 17 18:33:17.727295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.727233 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741"} err="failed to get container status \"97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741\": rpc error: code = NotFound desc = could not find container \"97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741\": container with ID starting with 97a9467a5863d5e21cebf3334c4f7ca1fbd21eb7af1ee466aeaf112827164741 not found: ID does not exist" Sep 17 18:33:17.735543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.735515 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9"] Sep 17 18:33:17.738380 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:17.738359 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/sub-test-model-95a92b-kserve-5d87787dd7-424w9"] Sep 17 18:33:19.346284 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:19.346250 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f759e79a-daf6-456c-a8e8-796c627f5789" path="/var/lib/kubelet/pods/f759e79a-daf6-456c-a8e8-796c627f5789/volumes" Sep 17 18:33:28.806168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.806124 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl"] Sep 17 18:33:28.807084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.806575 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f759e79a-daf6-456c-a8e8-796c627f5789" containerName="main" Sep 17 18:33:28.807084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.806586 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f759e79a-daf6-456c-a8e8-796c627f5789" containerName="main" Sep 17 18:33:28.807084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.806600 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="618538bf-c029-44ec-82e3-c83e0de9a4f2" containerName="test-curl-7631-e156fd" Sep 17 18:33:28.807084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.806606 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="618538bf-c029-44ec-82e3-c83e0de9a4f2" containerName="test-curl-7631-e156fd" Sep 17 18:33:28.807084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.806682 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f759e79a-daf6-456c-a8e8-796c627f5789" containerName="main" Sep 17 18:33:28.807084 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.806690 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="618538bf-c029-44ec-82e3-c83e0de9a4f2" containerName="test-curl-7631-e156fd" Sep 17 18:33:28.810380 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.810357 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:28.813736 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.813712 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"openshift-service-ca.crt\"" Sep 17 18:33:28.813885 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.813713 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"kube-root-ca.crt\"" Sep 17 18:33:28.813885 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.813755 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"rate-test-model-95a92b-kserve-self-signed-certs\"" Sep 17 18:33:28.813885 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.813714 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-95a92b\"/\"default-dockercfg-rf2c5\"" Sep 17 18:33:28.821876 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.821843 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl"] Sep 17 18:33:28.900707 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.900673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzh6\" (UniqueName: \"kubernetes.io/projected/94604fc4-51e4-4876-bcdf-eb389fc08609-kube-api-access-wxzh6\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:28.900924 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.900722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-tmp-dir\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:28.900924 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.900741 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-home\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:28.900924 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.900899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-model-cache\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:28.901080 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.900981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-dshm\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:28.901080 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:28.900998 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94604fc4-51e4-4876-bcdf-eb389fc08609-tls-certs\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.001994 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.001957 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-dshm\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.001994 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.001997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94604fc4-51e4-4876-bcdf-eb389fc08609-tls-certs\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.002201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.002021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzh6\" (UniqueName: \"kubernetes.io/projected/94604fc4-51e4-4876-bcdf-eb389fc08609-kube-api-access-wxzh6\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.002201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.002054 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-tmp-dir\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.002201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.002069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-home\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.002201 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.002114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-model-cache\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.002488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.002468 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-home\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.002546 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.002514 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-tmp-dir\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.002596 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.002516 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-model-cache\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.004280 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.004260 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-dshm\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.004601 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.004585 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94604fc4-51e4-4876-bcdf-eb389fc08609-tls-certs\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.009482 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.009459 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzh6\" (UniqueName: \"kubernetes.io/projected/94604fc4-51e4-4876-bcdf-eb389fc08609-kube-api-access-wxzh6\") pod \"rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.120136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.120062 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:29.396664 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.396638 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl"] Sep 17 18:33:29.398783 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:33:29.398752 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94604fc4_51e4_4876_bcdf_eb389fc08609.slice/crio-48f59c1b3ddf13bffcbe98f2d27046c6e30f9c1e60b29e3f6405bb2e8dfdf2c3 WatchSource:0}: Error finding container 48f59c1b3ddf13bffcbe98f2d27046c6e30f9c1e60b29e3f6405bb2e8dfdf2c3: Status 404 returned error can't find the container with id 48f59c1b3ddf13bffcbe98f2d27046c6e30f9c1e60b29e3f6405bb2e8dfdf2c3 Sep 17 18:33:29.786050 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.786006 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" event={"ID":"94604fc4-51e4-4876-bcdf-eb389fc08609","Type":"ContainerStarted","Data":"6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10"} Sep 17 18:33:29.786050 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.786055 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" event={"ID":"94604fc4-51e4-4876-bcdf-eb389fc08609","Type":"ContainerStarted","Data":"48f59c1b3ddf13bffcbe98f2d27046c6e30f9c1e60b29e3f6405bb2e8dfdf2c3"} Sep 17 18:33:29.803810 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:29.803745 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" podStartSLOduration=1.803718604 podStartE2EDuration="1.803718604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:29.803375311 +0000 UTC m=+1820.867319270" watchObservedRunningTime="2026-09-17 18:33:29.803718604 +0000 UTC m=+1820.867662569" Sep 17 18:33:39.120396 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:39.120348 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:39.120923 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:39.120520 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:39.132984 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:39.132963 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:33:39.853176 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:33:39.853149 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:34:39.580735 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:39.580648 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl"] Sep 17 18:34:46.769885 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.769834 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm"] Sep 17 18:34:46.776367 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.776341 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.779336 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.779313 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f72155-serving-cert\"" Sep 17 18:34:46.789027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.789006 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm"] Sep 17 18:34:46.857009 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.856976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.857167 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.857081 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4dmv\" (UniqueName: \"kubernetes.io/projected/53797075-f843-4f69-93e3-d795d301ff7d-kube-api-access-h4dmv\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.857167 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.857123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-metrics-certs\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.857167 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.857152 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53797075-f843-4f69-93e3-d795d301ff7d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.958024 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.957995 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4dmv\" (UniqueName: \"kubernetes.io/projected/53797075-f843-4f69-93e3-d795d301ff7d-kube-api-access-h4dmv\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.958186 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.958029 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-metrics-certs\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.958186 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.958062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53797075-f843-4f69-93e3-d795d301ff7d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.958186 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.958145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.958299 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:34:46.958260 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-serving-cert: secret "maas-api-e2e-ipp-a-f72155-serving-cert" not found Sep 17 18:34:46.958373 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:34:46.958362 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls podName:53797075-f843-4f69-93e3-d795d301ff7d nodeName:}" failed. No retries permitted until 2026-09-17 18:34:47.458339871 +0000 UTC m=+1898.522283810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls") pod "maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" (UID: "53797075-f843-4f69-93e3-d795d301ff7d") : secret "maas-api-e2e-ipp-a-f72155-serving-cert" not found Sep 17 18:34:46.958809 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.958788 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53797075-f843-4f69-93e3-d795d301ff7d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.960567 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.960546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-metrics-certs\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:46.967478 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:46.967458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4dmv\" (UniqueName: \"kubernetes.io/projected/53797075-f843-4f69-93e3-d795d301ff7d-kube-api-access-h4dmv\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:47.464001 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:47.463965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:47.466716 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:47.466687 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:47.688123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:47.688047 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:47.973338 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:47.973303 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm"] Sep 17 18:34:47.975416 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:34:47.975389 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53797075_f843_4f69_93e3_d795d301ff7d.slice/crio-29d37210495fefc69a22de126b3178757fad2ec7ba6a404cebefaa5cee0a188d WatchSource:0}: Error finding container 29d37210495fefc69a22de126b3178757fad2ec7ba6a404cebefaa5cee0a188d: Status 404 returned error can't find the container with id 29d37210495fefc69a22de126b3178757fad2ec7ba6a404cebefaa5cee0a188d Sep 17 18:34:47.976783 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:47.976766 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:34:48.222441 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:48.222405 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" event={"ID":"53797075-f843-4f69-93e3-d795d301ff7d","Type":"ContainerStarted","Data":"29d37210495fefc69a22de126b3178757fad2ec7ba6a404cebefaa5cee0a188d"} Sep 17 18:34:49.232938 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:49.232903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" event={"ID":"53797075-f843-4f69-93e3-d795d301ff7d","Type":"ContainerStarted","Data":"3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5"} Sep 17 18:34:49.233423 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:49.233103 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:49.252883 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:49.252814 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" podStartSLOduration=2.722088757 podStartE2EDuration="3.252793974s" totalImagesPullingTime="530.705217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:49.251936759 +0000 UTC m=+1900.315880719" watchObservedRunningTime="2026-09-17 18:34:49.252793974 +0000 UTC m=+1900.316737936" Sep 17 18:34:54.600387 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.600327 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" podUID="94604fc4-51e4-4876-bcdf-eb389fc08609" containerName="main" containerID="cri-o://6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10" gracePeriod=45 Sep 17 18:34:54.859416 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.859353 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:34:54.942492 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.942462 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94604fc4-51e4-4876-bcdf-eb389fc08609-tls-certs\") pod \"94604fc4-51e4-4876-bcdf-eb389fc08609\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " Sep 17 18:34:54.942669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.942568 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-tmp-dir\") pod \"94604fc4-51e4-4876-bcdf-eb389fc08609\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " Sep 17 18:34:54.942669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.942597 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxzh6\" (UniqueName: \"kubernetes.io/projected/94604fc4-51e4-4876-bcdf-eb389fc08609-kube-api-access-wxzh6\") pod \"94604fc4-51e4-4876-bcdf-eb389fc08609\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " Sep 17 18:34:54.942669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.942618 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-dshm\") pod \"94604fc4-51e4-4876-bcdf-eb389fc08609\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " Sep 17 18:34:54.942669 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.942642 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-home\") pod \"94604fc4-51e4-4876-bcdf-eb389fc08609\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " Sep 17 18:34:54.942932 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.942673 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-model-cache\") pod \"94604fc4-51e4-4876-bcdf-eb389fc08609\" (UID: \"94604fc4-51e4-4876-bcdf-eb389fc08609\") " Sep 17 18:34:54.942932 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.942837 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94604fc4-51e4-4876-bcdf-eb389fc08609" (UID: "94604fc4-51e4-4876-bcdf-eb389fc08609"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:34:54.943064 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.943007 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-home" (OuterVolumeSpecName: "home") pod "94604fc4-51e4-4876-bcdf-eb389fc08609" (UID: "94604fc4-51e4-4876-bcdf-eb389fc08609"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:34:54.943064 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.943058 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-model-cache" (OuterVolumeSpecName: "model-cache") pod "94604fc4-51e4-4876-bcdf-eb389fc08609" (UID: "94604fc4-51e4-4876-bcdf-eb389fc08609"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:34:54.943159 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.943140 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:34:54.943200 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.943162 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:34:54.944646 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.944614 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94604fc4-51e4-4876-bcdf-eb389fc08609-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94604fc4-51e4-4876-bcdf-eb389fc08609" (UID: "94604fc4-51e4-4876-bcdf-eb389fc08609"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:34:54.945125 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.945104 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-dshm" (OuterVolumeSpecName: "dshm") pod "94604fc4-51e4-4876-bcdf-eb389fc08609" (UID: "94604fc4-51e4-4876-bcdf-eb389fc08609"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:34:54.945214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:54.945140 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94604fc4-51e4-4876-bcdf-eb389fc08609-kube-api-access-wxzh6" (OuterVolumeSpecName: "kube-api-access-wxzh6") pod "94604fc4-51e4-4876-bcdf-eb389fc08609" (UID: "94604fc4-51e4-4876-bcdf-eb389fc08609"). InnerVolumeSpecName "kube-api-access-wxzh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:34:55.044192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.044149 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxzh6\" (UniqueName: \"kubernetes.io/projected/94604fc4-51e4-4876-bcdf-eb389fc08609-kube-api-access-wxzh6\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:34:55.044192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.044185 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:34:55.044192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.044194 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94604fc4-51e4-4876-bcdf-eb389fc08609-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:34:55.044192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.044204 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94604fc4-51e4-4876-bcdf-eb389fc08609-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:34:55.242955 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.242916 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:34:55.267397 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.267368 2582 generic.go:356] "Generic (PLEG): container finished" podID="94604fc4-51e4-4876-bcdf-eb389fc08609" containerID="6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10" exitCode=0 Sep 17 18:34:55.267545 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.267408 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" Sep 17 18:34:55.267545 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.267453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" event={"ID":"94604fc4-51e4-4876-bcdf-eb389fc08609","Type":"ContainerDied","Data":"6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10"} Sep 17 18:34:55.267545 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.267489 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl" event={"ID":"94604fc4-51e4-4876-bcdf-eb389fc08609","Type":"ContainerDied","Data":"48f59c1b3ddf13bffcbe98f2d27046c6e30f9c1e60b29e3f6405bb2e8dfdf2c3"} Sep 17 18:34:55.267545 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.267509 2582 scope.go:117] "RemoveContainer" containerID="6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10" Sep 17 18:34:55.280068 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.280042 2582 scope.go:117] "RemoveContainer" containerID="6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10" Sep 17 18:34:55.280703 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:34:55.280675 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10\": container with ID starting with 6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10 not found: ID does not exist" containerID="6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10" Sep 17 18:34:55.280805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.280715 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10"} err="failed to get container status \"6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10\": rpc error: code = NotFound desc = could not find container \"6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10\": container with ID starting with 6c9c7b35cb0652b01d67f664504a9fafb91e2aa3815ec283cdbfaa8ae81a9a10 not found: ID does not exist" Sep 17 18:34:55.291296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.291270 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl"] Sep 17 18:34:55.294806 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.294779 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-95a92b/rate-test-model-95a92b-kserve-6bd46d5bf5-q9xvl"] Sep 17 18:34:55.348839 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:34:55.348811 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94604fc4-51e4-4876-bcdf-eb389fc08609" path="/var/lib/kubelet/pods/94604fc4-51e4-4876-bcdf-eb389fc08609/volumes" Sep 17 18:35:17.107453 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.107420 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq"] Sep 17 18:35:17.108135 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.108115 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94604fc4-51e4-4876-bcdf-eb389fc08609" containerName="main" Sep 17 18:35:17.108135 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.108137 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="94604fc4-51e4-4876-bcdf-eb389fc08609" containerName="main" Sep 17 18:35:17.108295 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.108265 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="94604fc4-51e4-4876-bcdf-eb389fc08609" containerName="main" Sep 17 18:35:17.112667 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.112640 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.116101 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.116077 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0f4423-serving-cert\"" Sep 17 18:35:17.127130 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.127107 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq"] Sep 17 18:35:17.147918 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.147886 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a642e15c-f0bc-4580-a87a-5ac07bee1afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.148034 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.147960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.148133 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.148111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwkj\" (UniqueName: \"kubernetes.io/projected/a642e15c-f0bc-4580-a87a-5ac07bee1afa-kube-api-access-xwwkj\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.148186 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.148153 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-metrics-certs\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.249665 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.249633 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a642e15c-f0bc-4580-a87a-5ac07bee1afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.249665 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.249672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.249962 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.249759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwkj\" (UniqueName: \"kubernetes.io/projected/a642e15c-f0bc-4580-a87a-5ac07bee1afa-kube-api-access-xwwkj\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.249962 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.249794 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-metrics-certs\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.250365 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.250341 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a642e15c-f0bc-4580-a87a-5ac07bee1afa-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.252306 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.252280 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-metrics-certs\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.252418 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.252336 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.258644 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.258621 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwkj\" (UniqueName: \"kubernetes.io/projected/a642e15c-f0bc-4580-a87a-5ac07bee1afa-kube-api-access-xwwkj\") pod \"maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.425027 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.424997 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:17.717034 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:17.716999 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq"] Sep 17 18:35:17.718885 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:35:17.718825 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda642e15c_f0bc_4580_a87a_5ac07bee1afa.slice/crio-e099fa45137323a97be54d2a645687ca511d683d501facf705fd3f4040a16b67 WatchSource:0}: Error finding container e099fa45137323a97be54d2a645687ca511d683d501facf705fd3f4040a16b67: Status 404 returned error can't find the container with id e099fa45137323a97be54d2a645687ca511d683d501facf705fd3f4040a16b67 Sep 17 18:35:18.394744 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:18.394637 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" event={"ID":"a642e15c-f0bc-4580-a87a-5ac07bee1afa","Type":"ContainerStarted","Data":"7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499"} Sep 17 18:35:18.394744 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:18.394712 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:18.394744 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:18.394735 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" event={"ID":"a642e15c-f0bc-4580-a87a-5ac07bee1afa","Type":"ContainerStarted","Data":"e099fa45137323a97be54d2a645687ca511d683d501facf705fd3f4040a16b67"} Sep 17 18:35:18.414664 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:18.414618 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" podStartSLOduration=1.019834595 podStartE2EDuration="1.414603856s" totalImagesPullingTime="394.769261ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:18.412329954 +0000 UTC m=+1929.476273914" watchObservedRunningTime="2026-09-17 18:35:18.414603856 +0000 UTC m=+1929.478547814" Sep 17 18:35:24.404463 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:24.404431 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:35:44.502849 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:44.502817 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8k6rh_6927897e-3956-4585-8de2-bea9c1384dee/payload-processing/0.log" Sep 17 18:35:53.305462 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.305428 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r"] Sep 17 18:35:53.309733 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.309713 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.312368 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.312344 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f72155\"/\"openshift-service-ca.crt\"" Sep 17 18:35:53.312483 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.312404 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f72155\"/\"kube-root-ca.crt\"" Sep 17 18:35:53.313765 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.313742 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f72155\"/\"default-dockercfg-jzntc\"" Sep 17 18:35:53.313908 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.313789 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f72155\"/\"ipp-route-f72155-kserve-self-signed-certs\"" Sep 17 18:35:53.319620 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.319599 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r"] Sep 17 18:35:53.402770 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.402732 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-home\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.402770 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.402772 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-dshm\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.403010 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.402890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-model-cache\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.403010 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.402946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043aca49-aacc-4f1c-91c5-d8a987229b46-tls-certs\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.403081 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.403032 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-tmp-dir\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.403117 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.403103 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2m6x\" (UniqueName: \"kubernetes.io/projected/043aca49-aacc-4f1c-91c5-d8a987229b46-kube-api-access-q2m6x\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.503930 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.503891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2m6x\" (UniqueName: \"kubernetes.io/projected/043aca49-aacc-4f1c-91c5-d8a987229b46-kube-api-access-q2m6x\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.504105 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.503956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-home\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.504105 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.503991 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-dshm\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.504186 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.504139 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-model-cache\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.504186 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.504177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043aca49-aacc-4f1c-91c5-d8a987229b46-tls-certs\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.504277 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.504261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-tmp-dir\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.504410 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.504389 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-home\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.504496 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.504479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-model-cache\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.504549 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.504520 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-tmp-dir\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.506388 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.506367 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-dshm\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.506609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.506587 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043aca49-aacc-4f1c-91c5-d8a987229b46-tls-certs\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.512948 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.512919 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2m6x\" (UniqueName: \"kubernetes.io/projected/043aca49-aacc-4f1c-91c5-d8a987229b46-kube-api-access-q2m6x\") pod \"ipp-route-f72155-kserve-844f995947-69q5r\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.621210 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.621121 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:35:53.896013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:53.895983 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r"] Sep 17 18:35:53.898397 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:35:53.898357 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod043aca49_aacc_4f1c_91c5_d8a987229b46.slice/crio-08ec5a72ec3d3e8f2972b61777392a9074f00774d82a1123e49b369e00c6619f WatchSource:0}: Error finding container 08ec5a72ec3d3e8f2972b61777392a9074f00774d82a1123e49b369e00c6619f: Status 404 returned error can't find the container with id 08ec5a72ec3d3e8f2972b61777392a9074f00774d82a1123e49b369e00c6619f Sep 17 18:35:54.593099 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:54.593066 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" event={"ID":"043aca49-aacc-4f1c-91c5-d8a987229b46","Type":"ContainerStarted","Data":"0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464"} Sep 17 18:35:54.593467 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:54.593105 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" event={"ID":"043aca49-aacc-4f1c-91c5-d8a987229b46","Type":"ContainerStarted","Data":"08ec5a72ec3d3e8f2972b61777392a9074f00774d82a1123e49b369e00c6619f"} Sep 17 18:35:54.611220 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:35:54.611171 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" podStartSLOduration=1.6111555869999998 podStartE2EDuration="1.611155587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:54.609937897 +0000 UTC m=+1965.673881855" watchObservedRunningTime="2026-09-17 18:35:54.611155587 +0000 UTC m=+1965.675099606" Sep 17 18:36:03.622035 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:03.621996 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:36:03.622519 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:03.622058 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:36:03.634736 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:03.634711 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:36:03.654387 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:03.654358 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:36:17.798911 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:17.798854 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8k6rh_6927897e-3956-4585-8de2-bea9c1384dee/payload-processing/0.log" Sep 17 18:36:25.615029 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.614988 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2"] Sep 17 18:36:25.619983 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.619957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.622324 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.622295 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-c37358-serving-cert\"" Sep 17 18:36:25.628583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.628273 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.628583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.628327 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwsx\" (UniqueName: \"kubernetes.io/projected/03cd391a-0934-4847-9ce3-2e5ac7fbff98-kube-api-access-4nwsx\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.628583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.628345 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2"] Sep 17 18:36:25.628583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.628405 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03cd391a-0934-4847-9ce3-2e5ac7fbff98-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.628583 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.628513 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.729303 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.729265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.729500 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.729317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwsx\" (UniqueName: \"kubernetes.io/projected/03cd391a-0934-4847-9ce3-2e5ac7fbff98-kube-api-access-4nwsx\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.729500 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.729374 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03cd391a-0934-4847-9ce3-2e5ac7fbff98-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.729500 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:36:25.729422 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-serving-cert: secret "maas-api-e2e-ipp-cleanup-c37358-serving-cert" not found Sep 17 18:36:25.729500 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.729451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.729500 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:36:25.729487 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls podName:03cd391a-0934-4847-9ce3-2e5ac7fbff98 nodeName:}" failed. No retries permitted until 2026-09-17 18:36:26.229468626 +0000 UTC m=+1997.293412561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" (UID: "03cd391a-0934-4847-9ce3-2e5ac7fbff98") : secret "maas-api-e2e-ipp-cleanup-c37358-serving-cert" not found Sep 17 18:36:25.730330 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.730308 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03cd391a-0934-4847-9ce3-2e5ac7fbff98-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.732018 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.731998 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:25.737753 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:25.737730 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwsx\" (UniqueName: \"kubernetes.io/projected/03cd391a-0934-4847-9ce3-2e5ac7fbff98-kube-api-access-4nwsx\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:26.233527 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:26.233459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:26.236102 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:26.236078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:26.533452 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:26.533361 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:26.810947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:26.810918 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2"] Sep 17 18:36:26.813122 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:36:26.813094 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cd391a_0934_4847_9ce3_2e5ac7fbff98.slice/crio-10626901026aa1da6ac2df782d6750acb8ea69e635e7eee0e5321549cbf15bbc WatchSource:0}: Error finding container 10626901026aa1da6ac2df782d6750acb8ea69e635e7eee0e5321549cbf15bbc: Status 404 returned error can't find the container with id 10626901026aa1da6ac2df782d6750acb8ea69e635e7eee0e5321549cbf15bbc Sep 17 18:36:27.783337 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:27.783306 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" event={"ID":"03cd391a-0934-4847-9ce3-2e5ac7fbff98","Type":"ContainerStarted","Data":"d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1"} Sep 17 18:36:27.783523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:27.783364 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:36:27.783523 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:27.783382 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" event={"ID":"03cd391a-0934-4847-9ce3-2e5ac7fbff98","Type":"ContainerStarted","Data":"10626901026aa1da6ac2df782d6750acb8ea69e635e7eee0e5321549cbf15bbc"} Sep 17 18:36:27.800937 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:27.800889 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" podStartSLOduration=2.4553905670000002 podStartE2EDuration="2.800874574s" totalImagesPullingTime="345.484007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:27.799146651 +0000 UTC m=+1998.863090610" watchObservedRunningTime="2026-09-17 18:36:27.800874574 +0000 UTC m=+1998.864818524" Sep 17 18:36:33.794381 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:36:33.794347 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:37:03.890102 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:03.889664 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2"] Sep 17 18:37:03.890102 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:03.890017 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" podUID="03cd391a-0934-4847-9ce3-2e5ac7fbff98" containerName="maas-api" containerID="cri-o://d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1" gracePeriod=30 Sep 17 18:37:04.159000 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.158974 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:37:04.209963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.209935 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03cd391a-0934-4847-9ce3-2e5ac7fbff98-openshift-service-ca\") pod \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " Sep 17 18:37:04.210123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.210040 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-metrics-certs\") pod \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " Sep 17 18:37:04.210123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.210104 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls\") pod \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " Sep 17 18:37:04.210203 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.210132 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nwsx\" (UniqueName: \"kubernetes.io/projected/03cd391a-0934-4847-9ce3-2e5ac7fbff98-kube-api-access-4nwsx\") pod \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\" (UID: \"03cd391a-0934-4847-9ce3-2e5ac7fbff98\") " Sep 17 18:37:04.210408 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.210382 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cd391a-0934-4847-9ce3-2e5ac7fbff98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03cd391a-0934-4847-9ce3-2e5ac7fbff98" (UID: "03cd391a-0934-4847-9ce3-2e5ac7fbff98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:37:04.212296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.212266 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03cd391a-0934-4847-9ce3-2e5ac7fbff98" (UID: "03cd391a-0934-4847-9ce3-2e5ac7fbff98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:37:04.212466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.212444 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03cd391a-0934-4847-9ce3-2e5ac7fbff98" (UID: "03cd391a-0934-4847-9ce3-2e5ac7fbff98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:37:04.212527 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.212449 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cd391a-0934-4847-9ce3-2e5ac7fbff98-kube-api-access-4nwsx" (OuterVolumeSpecName: "kube-api-access-4nwsx") pod "03cd391a-0934-4847-9ce3-2e5ac7fbff98" (UID: "03cd391a-0934-4847-9ce3-2e5ac7fbff98"). InnerVolumeSpecName "kube-api-access-4nwsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:37:04.311472 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.311427 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:04.311472 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.311471 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nwsx\" (UniqueName: \"kubernetes.io/projected/03cd391a-0934-4847-9ce3-2e5ac7fbff98-kube-api-access-4nwsx\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:04.311729 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.311489 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03cd391a-0934-4847-9ce3-2e5ac7fbff98-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:04.311729 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:04.311503 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03cd391a-0934-4847-9ce3-2e5ac7fbff98-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:05.000421 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.000385 2582 generic.go:356] "Generic (PLEG): container finished" podID="03cd391a-0934-4847-9ce3-2e5ac7fbff98" containerID="d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1" exitCode=0 Sep 17 18:37:05.000924 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.000439 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" Sep 17 18:37:05.000924 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.000439 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" event={"ID":"03cd391a-0934-4847-9ce3-2e5ac7fbff98","Type":"ContainerDied","Data":"d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1"} Sep 17 18:37:05.000924 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.000535 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2" event={"ID":"03cd391a-0934-4847-9ce3-2e5ac7fbff98","Type":"ContainerDied","Data":"10626901026aa1da6ac2df782d6750acb8ea69e635e7eee0e5321549cbf15bbc"} Sep 17 18:37:05.000924 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.000567 2582 scope.go:117] "RemoveContainer" containerID="d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1" Sep 17 18:37:05.010992 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.010973 2582 scope.go:117] "RemoveContainer" containerID="d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1" Sep 17 18:37:05.011237 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:37:05.011219 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1\": container with ID starting with d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1 not found: ID does not exist" containerID="d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1" Sep 17 18:37:05.011293 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.011250 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1"} err="failed to get container status \"d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1\": rpc error: code = NotFound desc = could not find container \"d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1\": container with ID starting with d608c0e390cbcf96149f0f79d9dea032c061eb3e51933fd30ee4ae0b6f52a5b1 not found: ID does not exist" Sep 17 18:37:05.029156 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.029125 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2"] Sep 17 18:37:05.033466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.033440 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-c37358-696575d749-n9pw2"] Sep 17 18:37:05.350016 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:05.349926 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03cd391a-0934-4847-9ce3-2e5ac7fbff98" path="/var/lib/kubelet/pods/03cd391a-0934-4847-9ce3-2e5ac7fbff98/volumes" Sep 17 18:37:30.307947 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.307710 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm"] Sep 17 18:37:30.308768 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.308706 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" podUID="53797075-f843-4f69-93e3-d795d301ff7d" containerName="maas-api" containerID="cri-o://3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5" gracePeriod=30 Sep 17 18:37:30.572545 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.572467 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:37:30.683069 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.683033 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4dmv\" (UniqueName: \"kubernetes.io/projected/53797075-f843-4f69-93e3-d795d301ff7d-kube-api-access-h4dmv\") pod \"53797075-f843-4f69-93e3-d795d301ff7d\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " Sep 17 18:37:30.683232 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.683119 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-metrics-certs\") pod \"53797075-f843-4f69-93e3-d795d301ff7d\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " Sep 17 18:37:30.683314 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.683291 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls\") pod \"53797075-f843-4f69-93e3-d795d301ff7d\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " Sep 17 18:37:30.683398 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.683380 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53797075-f843-4f69-93e3-d795d301ff7d-openshift-service-ca\") pod \"53797075-f843-4f69-93e3-d795d301ff7d\" (UID: \"53797075-f843-4f69-93e3-d795d301ff7d\") " Sep 17 18:37:30.683803 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.683770 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53797075-f843-4f69-93e3-d795d301ff7d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "53797075-f843-4f69-93e3-d795d301ff7d" (UID: "53797075-f843-4f69-93e3-d795d301ff7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:37:30.685352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.685317 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "53797075-f843-4f69-93e3-d795d301ff7d" (UID: "53797075-f843-4f69-93e3-d795d301ff7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:37:30.685352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.685331 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "53797075-f843-4f69-93e3-d795d301ff7d" (UID: "53797075-f843-4f69-93e3-d795d301ff7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:37:30.685530 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.685398 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53797075-f843-4f69-93e3-d795d301ff7d-kube-api-access-h4dmv" (OuterVolumeSpecName: "kube-api-access-h4dmv") pod "53797075-f843-4f69-93e3-d795d301ff7d" (UID: "53797075-f843-4f69-93e3-d795d301ff7d"). InnerVolumeSpecName "kube-api-access-h4dmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:37:30.785015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.784974 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:30.785015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.785008 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53797075-f843-4f69-93e3-d795d301ff7d-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:30.785015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.785023 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53797075-f843-4f69-93e3-d795d301ff7d-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:30.785260 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:30.785036 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4dmv\" (UniqueName: \"kubernetes.io/projected/53797075-f843-4f69-93e3-d795d301ff7d-kube-api-access-h4dmv\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:31.149879 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.149818 2582 generic.go:356] "Generic (PLEG): container finished" podID="53797075-f843-4f69-93e3-d795d301ff7d" containerID="3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5" exitCode=0 Sep 17 18:37:31.150071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.149890 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" Sep 17 18:37:31.150071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.149906 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" event={"ID":"53797075-f843-4f69-93e3-d795d301ff7d","Type":"ContainerDied","Data":"3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5"} Sep 17 18:37:31.150071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.149963 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm" event={"ID":"53797075-f843-4f69-93e3-d795d301ff7d","Type":"ContainerDied","Data":"29d37210495fefc69a22de126b3178757fad2ec7ba6a404cebefaa5cee0a188d"} Sep 17 18:37:31.150071 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.149984 2582 scope.go:117] "RemoveContainer" containerID="3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5" Sep 17 18:37:31.161900 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.161858 2582 scope.go:117] "RemoveContainer" containerID="3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5" Sep 17 18:37:31.162214 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:37:31.162194 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5\": container with ID starting with 3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5 not found: ID does not exist" containerID="3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5" Sep 17 18:37:31.162308 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.162222 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5"} err="failed to get container status \"3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5\": rpc error: code = NotFound desc = could not find container \"3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5\": container with ID starting with 3e5175c160b777ddf64273891d387c8db709c10d2f16e90c41d0c08c3a2de4e5 not found: ID does not exist" Sep 17 18:37:31.181466 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.181437 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm"] Sep 17 18:37:31.188631 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.188608 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f72155-769d496bcc-bmsvm"] Sep 17 18:37:31.347221 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:31.347186 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53797075-f843-4f69-93e3-d795d301ff7d" path="/var/lib/kubelet/pods/53797075-f843-4f69-93e3-d795d301ff7d/volumes" Sep 17 18:37:35.945668 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:35.945630 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r"] Sep 17 18:37:50.965700 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:50.965661 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" podUID="043aca49-aacc-4f1c-91c5-d8a987229b46" containerName="main" containerID="cri-o://0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464" gracePeriod=45 Sep 17 18:37:51.230672 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.230615 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:37:51.274972 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.274931 2582 generic.go:356] "Generic (PLEG): container finished" podID="043aca49-aacc-4f1c-91c5-d8a987229b46" containerID="0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464" exitCode=0 Sep 17 18:37:51.275157 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.274992 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" Sep 17 18:37:51.275157 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.275044 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" event={"ID":"043aca49-aacc-4f1c-91c5-d8a987229b46","Type":"ContainerDied","Data":"0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464"} Sep 17 18:37:51.275157 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.275106 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r" event={"ID":"043aca49-aacc-4f1c-91c5-d8a987229b46","Type":"ContainerDied","Data":"08ec5a72ec3d3e8f2972b61777392a9074f00774d82a1123e49b369e00c6619f"} Sep 17 18:37:51.275157 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.275131 2582 scope.go:117] "RemoveContainer" containerID="0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464" Sep 17 18:37:51.279749 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.279711 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2m6x\" (UniqueName: \"kubernetes.io/projected/043aca49-aacc-4f1c-91c5-d8a987229b46-kube-api-access-q2m6x\") pod \"043aca49-aacc-4f1c-91c5-d8a987229b46\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " Sep 17 18:37:51.279893 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.279760 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-dshm\") pod \"043aca49-aacc-4f1c-91c5-d8a987229b46\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " Sep 17 18:37:51.279893 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.279794 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-tmp-dir\") pod \"043aca49-aacc-4f1c-91c5-d8a987229b46\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " Sep 17 18:37:51.280121 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.280105 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-home\") pod \"043aca49-aacc-4f1c-91c5-d8a987229b46\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " Sep 17 18:37:51.280210 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.280113 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "043aca49-aacc-4f1c-91c5-d8a987229b46" (UID: "043aca49-aacc-4f1c-91c5-d8a987229b46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:37:51.280210 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.280149 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043aca49-aacc-4f1c-91c5-d8a987229b46-tls-certs\") pod \"043aca49-aacc-4f1c-91c5-d8a987229b46\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " Sep 17 18:37:51.280316 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.280280 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-model-cache\") pod \"043aca49-aacc-4f1c-91c5-d8a987229b46\" (UID: \"043aca49-aacc-4f1c-91c5-d8a987229b46\") " Sep 17 18:37:51.280639 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.280620 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:51.280774 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.280652 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-model-cache" (OuterVolumeSpecName: "model-cache") pod "043aca49-aacc-4f1c-91c5-d8a987229b46" (UID: "043aca49-aacc-4f1c-91c5-d8a987229b46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:37:51.280933 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.280664 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-home" (OuterVolumeSpecName: "home") pod "043aca49-aacc-4f1c-91c5-d8a987229b46" (UID: "043aca49-aacc-4f1c-91c5-d8a987229b46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:37:51.282727 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.282703 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043aca49-aacc-4f1c-91c5-d8a987229b46-kube-api-access-q2m6x" (OuterVolumeSpecName: "kube-api-access-q2m6x") pod "043aca49-aacc-4f1c-91c5-d8a987229b46" (UID: "043aca49-aacc-4f1c-91c5-d8a987229b46"). InnerVolumeSpecName "kube-api-access-q2m6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:37:51.282989 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.282969 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043aca49-aacc-4f1c-91c5-d8a987229b46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "043aca49-aacc-4f1c-91c5-d8a987229b46" (UID: "043aca49-aacc-4f1c-91c5-d8a987229b46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:37:51.283079 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.282995 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-dshm" (OuterVolumeSpecName: "dshm") pod "043aca49-aacc-4f1c-91c5-d8a987229b46" (UID: "043aca49-aacc-4f1c-91c5-d8a987229b46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:37:51.285464 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.285443 2582 scope.go:117] "RemoveContainer" containerID="0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464" Sep 17 18:37:51.285812 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:37:51.285786 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464\": container with ID starting with 0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464 not found: ID does not exist" containerID="0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464" Sep 17 18:37:51.285903 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.285823 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464"} err="failed to get container status \"0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464\": rpc error: code = NotFound desc = could not find container \"0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464\": container with ID starting with 0acebdf28153c45c8db08fc669e886945b73cf91e110c3a431db8310e2aea464 not found: ID does not exist" Sep 17 18:37:51.381362 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.381334 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:51.381362 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.381362 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2m6x\" (UniqueName: \"kubernetes.io/projected/043aca49-aacc-4f1c-91c5-d8a987229b46-kube-api-access-q2m6x\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:51.381548 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.381378 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:51.381548 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.381392 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/043aca49-aacc-4f1c-91c5-d8a987229b46-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:51.381548 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.381403 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/043aca49-aacc-4f1c-91c5-d8a987229b46-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:37:51.591271 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.591195 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r"] Sep 17 18:37:51.597553 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:51.597523 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f72155/ipp-route-f72155-kserve-844f995947-69q5r"] Sep 17 18:37:53.345771 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:37:53.345740 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043aca49-aacc-4f1c-91c5-d8a987229b46" path="/var/lib/kubelet/pods/043aca49-aacc-4f1c-91c5-d8a987229b46/volumes" Sep 17 18:38:11.446995 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:11.446884 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:38:11.450081 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:11.450060 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:38:11.455805 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:11.455787 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:38:11.458987 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:11.458971 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:38:37.923053 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:37.923016 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq"] Sep 17 18:38:37.923557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:37.923310 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" podUID="a642e15c-f0bc-4580-a87a-5ac07bee1afa" containerName="maas-api" containerID="cri-o://7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499" gracePeriod=30 Sep 17 18:38:38.189283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.189217 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:38:38.325751 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.325723 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-maas-api-tls\") pod \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " Sep 17 18:38:38.325953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.325780 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a642e15c-f0bc-4580-a87a-5ac07bee1afa-openshift-service-ca\") pod \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " Sep 17 18:38:38.325953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.325842 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-metrics-certs\") pod \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " Sep 17 18:38:38.325953 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.325925 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwwkj\" (UniqueName: \"kubernetes.io/projected/a642e15c-f0bc-4580-a87a-5ac07bee1afa-kube-api-access-xwwkj\") pod \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\" (UID: \"a642e15c-f0bc-4580-a87a-5ac07bee1afa\") " Sep 17 18:38:38.326237 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.326210 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a642e15c-f0bc-4580-a87a-5ac07bee1afa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a642e15c-f0bc-4580-a87a-5ac07bee1afa" (UID: "a642e15c-f0bc-4580-a87a-5ac07bee1afa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:38:38.327950 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.327914 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a642e15c-f0bc-4580-a87a-5ac07bee1afa" (UID: "a642e15c-f0bc-4580-a87a-5ac07bee1afa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:38:38.328073 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.328004 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a642e15c-f0bc-4580-a87a-5ac07bee1afa" (UID: "a642e15c-f0bc-4580-a87a-5ac07bee1afa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:38:38.328138 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.328099 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a642e15c-f0bc-4580-a87a-5ac07bee1afa-kube-api-access-xwwkj" (OuterVolumeSpecName: "kube-api-access-xwwkj") pod "a642e15c-f0bc-4580-a87a-5ac07bee1afa" (UID: "a642e15c-f0bc-4580-a87a-5ac07bee1afa"). InnerVolumeSpecName "kube-api-access-xwwkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:38.427281 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.427250 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwwkj\" (UniqueName: \"kubernetes.io/projected/a642e15c-f0bc-4580-a87a-5ac07bee1afa-kube-api-access-xwwkj\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:38:38.427281 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.427276 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:38:38.427281 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.427286 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a642e15c-f0bc-4580-a87a-5ac07bee1afa-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:38:38.427556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.427294 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a642e15c-f0bc-4580-a87a-5ac07bee1afa-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:38:38.537576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.537490 2582 generic.go:356] "Generic (PLEG): container finished" podID="a642e15c-f0bc-4580-a87a-5ac07bee1afa" containerID="7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499" exitCode=0 Sep 17 18:38:38.537576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.537530 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" Sep 17 18:38:38.537576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.537561 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" event={"ID":"a642e15c-f0bc-4580-a87a-5ac07bee1afa","Type":"ContainerDied","Data":"7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499"} Sep 17 18:38:38.537816 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.537607 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq" event={"ID":"a642e15c-f0bc-4580-a87a-5ac07bee1afa","Type":"ContainerDied","Data":"e099fa45137323a97be54d2a645687ca511d683d501facf705fd3f4040a16b67"} Sep 17 18:38:38.537816 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.537627 2582 scope.go:117] "RemoveContainer" containerID="7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499" Sep 17 18:38:38.550251 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.550233 2582 scope.go:117] "RemoveContainer" containerID="7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499" Sep 17 18:38:38.550584 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:38:38.550566 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499\": container with ID starting with 7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499 not found: ID does not exist" containerID="7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499" Sep 17 18:38:38.550656 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.550591 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499"} err="failed to get container status \"7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499\": rpc error: code = NotFound desc = could not find container \"7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499\": container with ID starting with 7a5523b00710a27758d1f6e80a8ec864389bbe561336940ee90bf89a55101499 not found: ID does not exist" Sep 17 18:38:38.589486 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.589456 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq"] Sep 17 18:38:38.594647 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:38.594618 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0f4423-579dd6b4c9-9ctjq"] Sep 17 18:38:39.345421 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:39.345390 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a642e15c-f0bc-4580-a87a-5ac07bee1afa" path="/var/lib/kubelet/pods/a642e15c-f0bc-4580-a87a-5ac07bee1afa/volumes" Sep 17 18:38:56.523423 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.523388 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw"] Sep 17 18:38:56.523949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.523658 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" podUID="54f1fca9-8852-4882-b0c3-9cac6d64604b" containerName="maas-api" containerID="cri-o://6ac2e3059fd4c71f7d609d85222edabc17b489022f547d589c33e71f8c8b217e" gracePeriod=30 Sep 17 18:38:56.642282 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.642247 2582 generic.go:356] "Generic (PLEG): container finished" podID="54f1fca9-8852-4882-b0c3-9cac6d64604b" containerID="6ac2e3059fd4c71f7d609d85222edabc17b489022f547d589c33e71f8c8b217e" exitCode=0 Sep 17 18:38:56.642481 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.642412 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" event={"ID":"54f1fca9-8852-4882-b0c3-9cac6d64604b","Type":"ContainerDied","Data":"6ac2e3059fd4c71f7d609d85222edabc17b489022f547d589c33e71f8c8b217e"} Sep 17 18:38:56.786495 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.786437 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:38:56.898237 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.898206 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-maas-api-tls\") pod \"54f1fca9-8852-4882-b0c3-9cac6d64604b\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " Sep 17 18:38:56.898237 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.898246 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7lcd\" (UniqueName: \"kubernetes.io/projected/54f1fca9-8852-4882-b0c3-9cac6d64604b-kube-api-access-r7lcd\") pod \"54f1fca9-8852-4882-b0c3-9cac6d64604b\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " Sep 17 18:38:56.898462 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.898358 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f1fca9-8852-4882-b0c3-9cac6d64604b-openshift-service-ca\") pod \"54f1fca9-8852-4882-b0c3-9cac6d64604b\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " Sep 17 18:38:56.898462 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.898396 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-metrics-certs\") pod \"54f1fca9-8852-4882-b0c3-9cac6d64604b\" (UID: \"54f1fca9-8852-4882-b0c3-9cac6d64604b\") " Sep 17 18:38:56.898778 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.898750 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f1fca9-8852-4882-b0c3-9cac6d64604b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54f1fca9-8852-4882-b0c3-9cac6d64604b" (UID: "54f1fca9-8852-4882-b0c3-9cac6d64604b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:38:56.900381 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.900355 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54f1fca9-8852-4882-b0c3-9cac6d64604b" (UID: "54f1fca9-8852-4882-b0c3-9cac6d64604b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:38:56.900546 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.900529 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f1fca9-8852-4882-b0c3-9cac6d64604b-kube-api-access-r7lcd" (OuterVolumeSpecName: "kube-api-access-r7lcd") pod "54f1fca9-8852-4882-b0c3-9cac6d64604b" (UID: "54f1fca9-8852-4882-b0c3-9cac6d64604b"). InnerVolumeSpecName "kube-api-access-r7lcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:38:56.900594 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.900544 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "54f1fca9-8852-4882-b0c3-9cac6d64604b" (UID: "54f1fca9-8852-4882-b0c3-9cac6d64604b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:38:56.999357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.999325 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f1fca9-8852-4882-b0c3-9cac6d64604b-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:38:56.999357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.999351 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:38:56.999357 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.999360 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54f1fca9-8852-4882-b0c3-9cac6d64604b-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:38:56.999582 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:56.999368 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7lcd\" (UniqueName: \"kubernetes.io/projected/54f1fca9-8852-4882-b0c3-9cac6d64604b-kube-api-access-r7lcd\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:38:57.649946 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:57.649908 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" event={"ID":"54f1fca9-8852-4882-b0c3-9cac6d64604b","Type":"ContainerDied","Data":"e718894a86dbfb74ff6254683bad7874fd295d667391a5f26c7681eef19ae807"} Sep 17 18:38:57.649946 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:57.649931 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw" Sep 17 18:38:57.650407 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:57.649980 2582 scope.go:117] "RemoveContainer" containerID="6ac2e3059fd4c71f7d609d85222edabc17b489022f547d589c33e71f8c8b217e" Sep 17 18:38:57.697004 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:57.696967 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw"] Sep 17 18:38:57.715204 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:57.715172 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-95a92b-5688645944-9shkw"] Sep 17 18:38:59.348706 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:38:59.348669 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54f1fca9-8852-4882-b0c3-9cac6d64604b" path="/var/lib/kubelet/pods/54f1fca9-8852-4882-b0c3-9cac6d64604b/volumes" Sep 17 18:39:15.322152 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.322116 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p"] Sep 17 18:39:15.322981 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.322388 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" podUID="03f25ae0-cbc6-423a-b022-ed931bfc483e" containerName="maas-api" containerID="cri-o://cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3" gracePeriod=30 Sep 17 18:39:15.585078 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.585017 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:39:15.678768 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.678737 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f25ae0-cbc6-423a-b022-ed931bfc483e-openshift-service-ca\") pod \"03f25ae0-cbc6-423a-b022-ed931bfc483e\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " Sep 17 18:39:15.678768 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.678773 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-metrics-certs\") pod \"03f25ae0-cbc6-423a-b022-ed931bfc483e\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " Sep 17 18:39:15.679058 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.678905 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69mcp\" (UniqueName: \"kubernetes.io/projected/03f25ae0-cbc6-423a-b022-ed931bfc483e-kube-api-access-69mcp\") pod \"03f25ae0-cbc6-423a-b022-ed931bfc483e\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " Sep 17 18:39:15.679058 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.678960 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls\") pod \"03f25ae0-cbc6-423a-b022-ed931bfc483e\" (UID: \"03f25ae0-cbc6-423a-b022-ed931bfc483e\") " Sep 17 18:39:15.679279 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.679231 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03f25ae0-cbc6-423a-b022-ed931bfc483e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03f25ae0-cbc6-423a-b022-ed931bfc483e" (UID: "03f25ae0-cbc6-423a-b022-ed931bfc483e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:39:15.679439 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.679416 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03f25ae0-cbc6-423a-b022-ed931bfc483e-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:39:15.681289 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.681256 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03f25ae0-cbc6-423a-b022-ed931bfc483e" (UID: "03f25ae0-cbc6-423a-b022-ed931bfc483e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:39:15.681445 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.681429 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03f25ae0-cbc6-423a-b022-ed931bfc483e" (UID: "03f25ae0-cbc6-423a-b022-ed931bfc483e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:39:15.681726 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.681699 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f25ae0-cbc6-423a-b022-ed931bfc483e-kube-api-access-69mcp" (OuterVolumeSpecName: "kube-api-access-69mcp") pod "03f25ae0-cbc6-423a-b022-ed931bfc483e" (UID: "03f25ae0-cbc6-423a-b022-ed931bfc483e"). InnerVolumeSpecName "kube-api-access-69mcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:39:15.749535 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.749497 2582 generic.go:356] "Generic (PLEG): container finished" podID="03f25ae0-cbc6-423a-b022-ed931bfc483e" containerID="cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3" exitCode=0 Sep 17 18:39:15.749726 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.749556 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" Sep 17 18:39:15.749726 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.749581 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" event={"ID":"03f25ae0-cbc6-423a-b022-ed931bfc483e","Type":"ContainerDied","Data":"cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3"} Sep 17 18:39:15.749726 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.749650 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p" event={"ID":"03f25ae0-cbc6-423a-b022-ed931bfc483e","Type":"ContainerDied","Data":"dca44555d14db632202086fad153531da50f9e10c48ed1cc606775b1507bc429"} Sep 17 18:39:15.749726 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.749682 2582 scope.go:117] "RemoveContainer" containerID="cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3" Sep 17 18:39:15.760895 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.760856 2582 scope.go:117] "RemoveContainer" containerID="cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3" Sep 17 18:39:15.761167 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:39:15.761146 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3\": container with ID starting with cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3 not found: ID does not exist" containerID="cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3" Sep 17 18:39:15.761234 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.761178 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3"} err="failed to get container status \"cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3\": rpc error: code = NotFound desc = could not find container \"cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3\": container with ID starting with cd2b0832080d46a13486391c8d0ca71b53d6362c1f008b7881490a65c2ec42e3 not found: ID does not exist" Sep 17 18:39:15.778043 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.778010 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p"] Sep 17 18:39:15.780160 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.780141 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69mcp\" (UniqueName: \"kubernetes.io/projected/03f25ae0-cbc6-423a-b022-ed931bfc483e-kube-api-access-69mcp\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:39:15.780227 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.780166 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:39:15.780227 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.780176 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03f25ae0-cbc6-423a-b022-ed931bfc483e-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:39:15.787820 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:15.787797 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-d81d7c-6bb5c97d78-dww4p"] Sep 17 18:39:17.346763 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:17.346726 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03f25ae0-cbc6-423a-b022-ed931bfc483e" path="/var/lib/kubelet/pods/03f25ae0-cbc6-423a-b022-ed931bfc483e/volumes" Sep 17 18:39:27.606796 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.606760 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-b86fq"] Sep 17 18:39:27.607325 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.607039 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c8bff4649-b86fq" podUID="fbb88877-0e7e-443d-afb5-89d9047e0bbf" containerName="manager" containerID="cri-o://e75ee19deeaa017f104832ed552976a2c6d343750bc4849e740c995a50731ecb" gracePeriod=30 Sep 17 18:39:27.818852 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.818816 2582 generic.go:356] "Generic (PLEG): container finished" podID="fbb88877-0e7e-443d-afb5-89d9047e0bbf" containerID="e75ee19deeaa017f104832ed552976a2c6d343750bc4849e740c995a50731ecb" exitCode=0 Sep 17 18:39:27.819026 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.818887 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-b86fq" event={"ID":"fbb88877-0e7e-443d-afb5-89d9047e0bbf","Type":"ContainerDied","Data":"e75ee19deeaa017f104832ed552976a2c6d343750bc4849e740c995a50731ecb"} Sep 17 18:39:27.861456 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.861402 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:39:27.995571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.995532 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-webhook-cert\") pod \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " Sep 17 18:39:27.995763 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.995663 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hvxp\" (UniqueName: \"kubernetes.io/projected/fbb88877-0e7e-443d-afb5-89d9047e0bbf-kube-api-access-2hvxp\") pod \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " Sep 17 18:39:27.995763 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.995711 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-metrics-certs\") pod \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\" (UID: \"fbb88877-0e7e-443d-afb5-89d9047e0bbf\") " Sep 17 18:39:27.997913 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.997880 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb88877-0e7e-443d-afb5-89d9047e0bbf-kube-api-access-2hvxp" (OuterVolumeSpecName: "kube-api-access-2hvxp") pod "fbb88877-0e7e-443d-afb5-89d9047e0bbf" (UID: "fbb88877-0e7e-443d-afb5-89d9047e0bbf"). InnerVolumeSpecName "kube-api-access-2hvxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:39:27.997913 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.997901 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fbb88877-0e7e-443d-afb5-89d9047e0bbf" (UID: "fbb88877-0e7e-443d-afb5-89d9047e0bbf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:39:27.998090 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:27.998040 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fbb88877-0e7e-443d-afb5-89d9047e0bbf" (UID: "fbb88877-0e7e-443d-afb5-89d9047e0bbf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:39:28.096790 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:28.096764 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hvxp\" (UniqueName: \"kubernetes.io/projected/fbb88877-0e7e-443d-afb5-89d9047e0bbf-kube-api-access-2hvxp\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.096790 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:28.096790 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.096981 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:28.096799 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbb88877-0e7e-443d-afb5-89d9047e0bbf-webhook-cert\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:39:28.825611 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:28.825576 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-b86fq" event={"ID":"fbb88877-0e7e-443d-afb5-89d9047e0bbf","Type":"ContainerDied","Data":"76a459bad06a16031f787313dbcaade9bcd6a3662280de1b6e976608d5b54962"} Sep 17 18:39:28.825611 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:28.825598 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-b86fq" Sep 17 18:39:28.826073 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:28.825643 2582 scope.go:117] "RemoveContainer" containerID="e75ee19deeaa017f104832ed552976a2c6d343750bc4849e740c995a50731ecb" Sep 17 18:39:28.847151 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:28.847128 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-b86fq"] Sep 17 18:39:28.851039 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:28.851016 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-b86fq"] Sep 17 18:39:29.347100 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:39:29.347064 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb88877-0e7e-443d-afb5-89d9047e0bbf" path="/var/lib/kubelet/pods/fbb88877-0e7e-443d-afb5-89d9047e0bbf/volumes" Sep 17 18:40:29.671291 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671199 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c8bff4649-cll9l"] Sep 17 18:40:29.671835 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671818 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03cd391a-0934-4847-9ce3-2e5ac7fbff98" containerName="maas-api" Sep 17 18:40:29.671949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671839 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03cd391a-0934-4847-9ce3-2e5ac7fbff98" containerName="maas-api" Sep 17 18:40:29.671949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671850 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54f1fca9-8852-4882-b0c3-9cac6d64604b" containerName="maas-api" Sep 17 18:40:29.671949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671859 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="54f1fca9-8852-4882-b0c3-9cac6d64604b" containerName="maas-api" Sep 17 18:40:29.671949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671905 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb88877-0e7e-443d-afb5-89d9047e0bbf" containerName="manager" Sep 17 18:40:29.671949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671915 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb88877-0e7e-443d-afb5-89d9047e0bbf" containerName="manager" Sep 17 18:40:29.671949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671943 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043aca49-aacc-4f1c-91c5-d8a987229b46" containerName="main" Sep 17 18:40:29.671949 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671952 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="043aca49-aacc-4f1c-91c5-d8a987229b46" containerName="main" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671971 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53797075-f843-4f69-93e3-d795d301ff7d" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671979 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="53797075-f843-4f69-93e3-d795d301ff7d" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671990 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a642e15c-f0bc-4580-a87a-5ac07bee1afa" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.671997 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a642e15c-f0bc-4580-a87a-5ac07bee1afa" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672011 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f25ae0-cbc6-423a-b022-ed931bfc483e" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672019 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f25ae0-cbc6-423a-b022-ed931bfc483e" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672118 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03cd391a-0934-4847-9ce3-2e5ac7fbff98" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672131 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="53797075-f843-4f69-93e3-d795d301ff7d" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672143 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="54f1fca9-8852-4882-b0c3-9cac6d64604b" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672152 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a642e15c-f0bc-4580-a87a-5ac07bee1afa" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672164 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb88877-0e7e-443d-afb5-89d9047e0bbf" containerName="manager" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672172 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f25ae0-cbc6-423a-b022-ed931bfc483e" containerName="maas-api" Sep 17 18:40:29.672756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.672183 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="043aca49-aacc-4f1c-91c5-d8a987229b46" containerName="main" Sep 17 18:40:29.677099 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.677080 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.680421 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.680399 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6gb7f\"" Sep 17 18:40:29.680543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.680435 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 18:40:29.680543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.680399 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 18:40:29.680543 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.680491 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 18:40:29.686459 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.686437 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-cll9l"] Sep 17 18:40:29.774194 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.774163 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-metrics-certs\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.774342 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.774228 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-webhook-cert\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.774342 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.774301 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skd6f\" (UniqueName: \"kubernetes.io/projected/2973f0f6-3919-4e79-9c4f-8e5982eab100-kube-api-access-skd6f\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.875626 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.875598 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-metrics-certs\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.875766 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.875662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-webhook-cert\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.875766 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.875720 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skd6f\" (UniqueName: \"kubernetes.io/projected/2973f0f6-3919-4e79-9c4f-8e5982eab100-kube-api-access-skd6f\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.877959 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.877930 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-webhook-cert\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.877959 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.877949 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-metrics-certs\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.884009 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.883986 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skd6f\" (UniqueName: \"kubernetes.io/projected/2973f0f6-3919-4e79-9c4f-8e5982eab100-kube-api-access-skd6f\") pod \"maas-controller-7c8bff4649-cll9l\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:29.988088 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:29.988017 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:30.263131 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:30.263103 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-cll9l"] Sep 17 18:40:30.265577 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:40:30.265538 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2973f0f6_3919_4e79_9c4f_8e5982eab100.slice/crio-2dfbc79840a1f69684f97b16434dd9d494434b610769bf69c58214f6b5737350 WatchSource:0}: Error finding container 2dfbc79840a1f69684f97b16434dd9d494434b610769bf69c58214f6b5737350: Status 404 returned error can't find the container with id 2dfbc79840a1f69684f97b16434dd9d494434b610769bf69c58214f6b5737350 Sep 17 18:40:30.266966 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:30.266945 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:40:31.153617 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:31.153580 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-cll9l" event={"ID":"2973f0f6-3919-4e79-9c4f-8e5982eab100","Type":"ContainerStarted","Data":"57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41"} Sep 17 18:40:31.154119 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:31.153647 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:31.154119 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:31.153669 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-cll9l" event={"ID":"2973f0f6-3919-4e79-9c4f-8e5982eab100","Type":"ContainerStarted","Data":"2dfbc79840a1f69684f97b16434dd9d494434b610769bf69c58214f6b5737350"} Sep 17 18:40:31.172666 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:31.172621 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c8bff4649-cll9l" podStartSLOduration=1.5637802299999999 podStartE2EDuration="2.172603992s" totalImagesPullingTime="608.823762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:31.171257437 +0000 UTC m=+2242.235201408" watchObservedRunningTime="2026-09-17 18:40:31.172603992 +0000 UTC m=+2242.236547949" Sep 17 18:40:42.162188 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:42.162150 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:53.623660 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.623621 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-cll9l"] Sep 17 18:40:53.624109 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.623826 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c8bff4649-cll9l" podUID="2973f0f6-3919-4e79-9c4f-8e5982eab100" containerName="manager" containerID="cri-o://57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41" gracePeriod=30 Sep 17 18:40:53.882556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.882498 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:53.989409 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.989379 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-metrics-certs\") pod \"2973f0f6-3919-4e79-9c4f-8e5982eab100\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " Sep 17 18:40:53.989595 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.989433 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-webhook-cert\") pod \"2973f0f6-3919-4e79-9c4f-8e5982eab100\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " Sep 17 18:40:53.989595 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.989488 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skd6f\" (UniqueName: \"kubernetes.io/projected/2973f0f6-3919-4e79-9c4f-8e5982eab100-kube-api-access-skd6f\") pod \"2973f0f6-3919-4e79-9c4f-8e5982eab100\" (UID: \"2973f0f6-3919-4e79-9c4f-8e5982eab100\") " Sep 17 18:40:53.991433 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.991399 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2973f0f6-3919-4e79-9c4f-8e5982eab100" (UID: "2973f0f6-3919-4e79-9c4f-8e5982eab100"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:40:53.991559 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.991442 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2973f0f6-3919-4e79-9c4f-8e5982eab100" (UID: "2973f0f6-3919-4e79-9c4f-8e5982eab100"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:40:53.991559 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:53.991455 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2973f0f6-3919-4e79-9c4f-8e5982eab100-kube-api-access-skd6f" (OuterVolumeSpecName: "kube-api-access-skd6f") pod "2973f0f6-3919-4e79-9c4f-8e5982eab100" (UID: "2973f0f6-3919-4e79-9c4f-8e5982eab100"). InnerVolumeSpecName "kube-api-access-skd6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:40:54.091168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.091141 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:40:54.091168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.091166 2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2973f0f6-3919-4e79-9c4f-8e5982eab100-webhook-cert\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:40:54.091337 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.091175 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skd6f\" (UniqueName: \"kubernetes.io/projected/2973f0f6-3919-4e79-9c4f-8e5982eab100-kube-api-access-skd6f\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:40:54.277307 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.277269 2582 generic.go:356] "Generic (PLEG): container finished" podID="2973f0f6-3919-4e79-9c4f-8e5982eab100" containerID="57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41" exitCode=0 Sep 17 18:40:54.277465 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.277319 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-cll9l" Sep 17 18:40:54.277465 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.277339 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-cll9l" event={"ID":"2973f0f6-3919-4e79-9c4f-8e5982eab100","Type":"ContainerDied","Data":"57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41"} Sep 17 18:40:54.277465 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.277427 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-cll9l" event={"ID":"2973f0f6-3919-4e79-9c4f-8e5982eab100","Type":"ContainerDied","Data":"2dfbc79840a1f69684f97b16434dd9d494434b610769bf69c58214f6b5737350"} Sep 17 18:40:54.277465 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.277457 2582 scope.go:117] "RemoveContainer" containerID="57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41" Sep 17 18:40:54.287112 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.287093 2582 scope.go:117] "RemoveContainer" containerID="57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41" Sep 17 18:40:54.287411 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:40:54.287387 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41\": container with ID starting with 57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41 not found: ID does not exist" containerID="57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41" Sep 17 18:40:54.287484 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.287425 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41"} err="failed to get container status \"57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41\": rpc error: code = NotFound desc = could not find container \"57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41\": container with ID starting with 57a644e0fadea1ee2706d0e517e4f62cf03dcd8f19d40c5bfe1c9e0ef7da5f41 not found: ID does not exist" Sep 17 18:40:54.302997 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.302975 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-cll9l"] Sep 17 18:40:54.308885 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:54.308845 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-cll9l"] Sep 17 18:40:55.088142 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.088109 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c8bff4649-k7g4p"] Sep 17 18:40:55.088569 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.088555 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2973f0f6-3919-4e79-9c4f-8e5982eab100" containerName="manager" Sep 17 18:40:55.088610 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.088572 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2973f0f6-3919-4e79-9c4f-8e5982eab100" containerName="manager" Sep 17 18:40:55.088693 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.088680 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2973f0f6-3919-4e79-9c4f-8e5982eab100" containerName="manager" Sep 17 18:40:55.093206 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.093187 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.096539 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.096510 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6gb7f\"" Sep 17 18:40:55.096652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.096552 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 17 18:40:55.096652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.096565 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 17 18:40:55.096652 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.096569 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 17 18:40:55.104778 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.104757 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-k7g4p"] Sep 17 18:40:55.203720 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.203689 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggr6\" (UniqueName: \"kubernetes.io/projected/846f3c5d-7aec-47b2-a72e-3006131207be-kube-api-access-5ggr6\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.203904 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.203726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846f3c5d-7aec-47b2-a72e-3006131207be-metrics-certs\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.203904 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.203819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846f3c5d-7aec-47b2-a72e-3006131207be-webhook-cert\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.305126 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.305084 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846f3c5d-7aec-47b2-a72e-3006131207be-webhook-cert\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.305292 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.305194 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggr6\" (UniqueName: \"kubernetes.io/projected/846f3c5d-7aec-47b2-a72e-3006131207be-kube-api-access-5ggr6\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.305292 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.305222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846f3c5d-7aec-47b2-a72e-3006131207be-metrics-certs\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.307581 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.307559 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846f3c5d-7aec-47b2-a72e-3006131207be-metrics-certs\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.307657 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.307584 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/846f3c5d-7aec-47b2-a72e-3006131207be-webhook-cert\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.315297 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.315274 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggr6\" (UniqueName: \"kubernetes.io/projected/846f3c5d-7aec-47b2-a72e-3006131207be-kube-api-access-5ggr6\") pod \"maas-controller-7c8bff4649-k7g4p\" (UID: \"846f3c5d-7aec-47b2-a72e-3006131207be\") " pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.346312 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.346249 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2973f0f6-3919-4e79-9c4f-8e5982eab100" path="/var/lib/kubelet/pods/2973f0f6-3919-4e79-9c4f-8e5982eab100/volumes" Sep 17 18:40:55.404428 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.404390 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:55.677168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:55.677003 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c8bff4649-k7g4p"] Sep 17 18:40:55.680008 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:40:55.679971 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846f3c5d_7aec_47b2_a72e_3006131207be.slice/crio-c0ab69018685415666732fcf4a3a9576e7e54d6990a9eb89ac45112d75189324 WatchSource:0}: Error finding container c0ab69018685415666732fcf4a3a9576e7e54d6990a9eb89ac45112d75189324: Status 404 returned error can't find the container with id c0ab69018685415666732fcf4a3a9576e7e54d6990a9eb89ac45112d75189324 Sep 17 18:40:56.291790 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:56.291753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-k7g4p" event={"ID":"846f3c5d-7aec-47b2-a72e-3006131207be","Type":"ContainerStarted","Data":"b308998b32eddf2b087e8e5c29b529b41afedf09d83e8cbc4dd583b566621633"} Sep 17 18:40:56.292225 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:56.291810 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c8bff4649-k7g4p" event={"ID":"846f3c5d-7aec-47b2-a72e-3006131207be","Type":"ContainerStarted","Data":"c0ab69018685415666732fcf4a3a9576e7e54d6990a9eb89ac45112d75189324"} Sep 17 18:40:56.292225 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:56.291895 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:40:56.312406 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:40:56.312356 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c8bff4649-k7g4p" podStartSLOduration=0.869487668 podStartE2EDuration="1.312340411s" totalImagesPullingTime="442.852743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:40:56.310326186 +0000 UTC m=+2267.374270146" watchObservedRunningTime="2026-09-17 18:40:56.312340411 +0000 UTC m=+2267.376284369" Sep 17 18:41:07.299879 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:41:07.299826 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c8bff4649-k7g4p" Sep 17 18:42:12.450541 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:12.450503 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk"] Sep 17 18:42:12.451029 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:12.450689 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" podUID="8afe7ab4-a326-4fb6-b9b5-c894e59c317b" containerName="manager" containerID="cri-o://fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683" gracePeriod=10 Sep 17 18:42:13.607590 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.607567 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:42:13.710841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.710744 2582 generic.go:356] "Generic (PLEG): container finished" podID="8afe7ab4-a326-4fb6-b9b5-c894e59c317b" containerID="fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683" exitCode=0 Sep 17 18:42:13.710841 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.710788 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" Sep 17 18:42:13.711074 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.710831 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" event={"ID":"8afe7ab4-a326-4fb6-b9b5-c894e59c317b","Type":"ContainerDied","Data":"fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683"} Sep 17 18:42:13.711074 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.710925 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk" event={"ID":"8afe7ab4-a326-4fb6-b9b5-c894e59c317b","Type":"ContainerDied","Data":"83ff4f7220cf18a2eaa4f7b9808b968a718eff812c0bee91d3185dec4d056aff"} Sep 17 18:42:13.711074 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.710959 2582 scope.go:117] "RemoveContainer" containerID="fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683" Sep 17 18:42:13.722722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.722695 2582 scope.go:117] "RemoveContainer" containerID="fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683" Sep 17 18:42:13.723179 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:42:13.723159 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683\": container with ID starting with fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683 not found: ID does not exist" containerID="fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683" Sep 17 18:42:13.723270 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.723191 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683"} err="failed to get container status \"fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683\": rpc error: code = NotFound desc = could not find container \"fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683\": container with ID starting with fcce2b4a8b77d775dd381ca33ebd1aea1db4ea794160469064162d653c3da683 not found: ID does not exist" Sep 17 18:42:13.753602 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.753578 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-extensions-socket-volume\") pod \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\" (UID: \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\") " Sep 17 18:42:13.753698 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.753616 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tcth\" (UniqueName: \"kubernetes.io/projected/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-kube-api-access-5tcth\") pod \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\" (UID: \"8afe7ab4-a326-4fb6-b9b5-c894e59c317b\") " Sep 17 18:42:13.754012 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.753993 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8afe7ab4-a326-4fb6-b9b5-c894e59c317b" (UID: "8afe7ab4-a326-4fb6-b9b5-c894e59c317b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:42:13.755614 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.755593 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-kube-api-access-5tcth" (OuterVolumeSpecName: "kube-api-access-5tcth") pod "8afe7ab4-a326-4fb6-b9b5-c894e59c317b" (UID: "8afe7ab4-a326-4fb6-b9b5-c894e59c317b"). InnerVolumeSpecName "kube-api-access-5tcth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:42:13.854562 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.854526 2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-extensions-socket-volume\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:42:13.854562 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:13.854557 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tcth\" (UniqueName: \"kubernetes.io/projected/8afe7ab4-a326-4fb6-b9b5-c894e59c317b-kube-api-access-5tcth\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:42:14.038068 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:14.038036 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk"] Sep 17 18:42:14.042601 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:14.042579 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w2frk"] Sep 17 18:42:15.345647 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:42:15.345606 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8afe7ab4-a326-4fb6-b9b5-c894e59c317b" path="/var/lib/kubelet/pods/8afe7ab4-a326-4fb6-b9b5-c894e59c317b/volumes" Sep 17 18:43:11.484451 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:11.484354 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:43:11.488618 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:11.487435 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:43:11.489225 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:11.489202 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:43:11.492273 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:11.492255 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kqs7r_d9b01902-a855-4aec-8617-33e5e2ab66ea/ovn-acl-logging/0.log" Sep 17 18:43:13.150028 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.149989 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2"] Sep 17 18:43:13.150561 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.150543 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afe7ab4-a326-4fb6-b9b5-c894e59c317b" containerName="manager" Sep 17 18:43:13.150561 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.150562 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afe7ab4-a326-4fb6-b9b5-c894e59c317b" containerName="manager" Sep 17 18:43:13.150636 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.150631 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afe7ab4-a326-4fb6-b9b5-c894e59c317b" containerName="manager" Sep 17 18:43:13.154230 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.154214 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:13.157061 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.157045 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zkxj7\"" Sep 17 18:43:13.171405 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.171381 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2"] Sep 17 18:43:13.293671 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.293634 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lq7l2\" (UID: \"3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:13.293893 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.293684 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkbf\" (UniqueName: \"kubernetes.io/projected/3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5-kube-api-access-wlkbf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lq7l2\" (UID: \"3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:13.394431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.394396 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lq7l2\" (UID: \"3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:13.394574 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.394441 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkbf\" (UniqueName: \"kubernetes.io/projected/3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5-kube-api-access-wlkbf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lq7l2\" (UID: \"3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:13.394767 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.394745 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lq7l2\" (UID: \"3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:13.404739 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.404673 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkbf\" (UniqueName: \"kubernetes.io/projected/3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5-kube-api-access-wlkbf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lq7l2\" (UID: \"3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:13.464527 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.464500 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:13.756232 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:13.756153 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2"] Sep 17 18:43:13.759914 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:43:13.759886 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c41efb9_0922_4c55_b7c0_8ec27cdcd8c5.slice/crio-eff978e4e3b7b1b9c6ea1d74b26ac4390ace2506570511d7b4d70726c744395d WatchSource:0}: Error finding container eff978e4e3b7b1b9c6ea1d74b26ac4390ace2506570511d7b4d70726c744395d: Status 404 returned error can't find the container with id eff978e4e3b7b1b9c6ea1d74b26ac4390ace2506570511d7b4d70726c744395d Sep 17 18:43:14.037514 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:14.037429 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" event={"ID":"3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5","Type":"ContainerStarted","Data":"911dd0293b91d4fb015b474c2f054bea850222936d82a400f15101b799a011f0"} Sep 17 18:43:14.037514 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:14.037481 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" event={"ID":"3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5","Type":"ContainerStarted","Data":"eff978e4e3b7b1b9c6ea1d74b26ac4390ace2506570511d7b4d70726c744395d"} Sep 17 18:43:14.037704 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:14.037520 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:43:14.066190 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:14.066141 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" podStartSLOduration=1.066125393 podStartE2EDuration="1.066125393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:43:14.063582976 +0000 UTC m=+2405.127526933" watchObservedRunningTime="2026-09-17 18:43:14.066125393 +0000 UTC m=+2405.130069351" Sep 17 18:43:25.046192 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:43:25.046120 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lq7l2" Sep 17 18:44:06.609431 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.609393 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj"] Sep 17 18:44:06.613471 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.613448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.616111 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.616091 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-79ed8f9a-kserve-self-signed-certs\"" Sep 17 18:44:06.622036 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.622014 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj"] Sep 17 18:44:06.703013 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.702970 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tls-certs\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.703207 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.703033 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-home\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.703207 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.703052 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-model-cache\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.703207 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.703120 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-dshm\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.703207 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.703165 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tmp-dir\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.703452 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.703263 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklhm\" (UniqueName: \"kubernetes.io/projected/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-kube-api-access-dklhm\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804274 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-home\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804304 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-model-cache\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804536 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804510 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-dshm\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804619 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804606 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tmp-dir\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804685 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804646 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-home\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804685 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklhm\" (UniqueName: \"kubernetes.io/projected/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-kube-api-access-dklhm\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804786 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-model-cache\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804786 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804729 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tls-certs\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.804979 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.804961 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tmp-dir\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.806750 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.806725 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-dshm\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.807124 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.807105 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tls-certs\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.823723 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.823693 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklhm\" (UniqueName: \"kubernetes.io/projected/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-kube-api-access-dklhm\") pod \"e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:06.924120 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:06.924089 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:07.205852 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:07.205827 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj"] Sep 17 18:44:07.207783 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:44:07.207756 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc79673c9_eaa5_499e_a4f6_4309ef5dfa02.slice/crio-0ecd62e954d94743e90caf6fa9b4e49938d6a98d5874330cb838b7484487fc8e WatchSource:0}: Error finding container 0ecd62e954d94743e90caf6fa9b4e49938d6a98d5874330cb838b7484487fc8e: Status 404 returned error can't find the container with id 0ecd62e954d94743e90caf6fa9b4e49938d6a98d5874330cb838b7484487fc8e Sep 17 18:44:07.348135 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:07.348088 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" event={"ID":"c79673c9-eaa5-499e-a4f6-4309ef5dfa02","Type":"ContainerStarted","Data":"ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace"} Sep 17 18:44:07.348135 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:07.348137 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" event={"ID":"c79673c9-eaa5-499e-a4f6-4309ef5dfa02","Type":"ContainerStarted","Data":"0ecd62e954d94743e90caf6fa9b4e49938d6a98d5874330cb838b7484487fc8e"} Sep 17 18:44:07.367386 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:07.367333 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" podStartSLOduration=1.367316156 podStartE2EDuration="1.367316156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:07.365531925 +0000 UTC m=+2458.429475884" watchObservedRunningTime="2026-09-17 18:44:07.367316156 +0000 UTC m=+2458.431260114" Sep 17 18:44:16.924636 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:16.924590 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:16.925103 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:16.924656 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:16.937257 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:16.937224 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:17.413105 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:17.413078 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:44:20.098556 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.098524 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj"] Sep 17 18:44:20.102611 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.102586 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.105341 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.105320 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-79ed8f9a-kserve-self-signed-certs\"" Sep 17 18:44:20.112062 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.112039 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj"] Sep 17 18:44:20.245126 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.245085 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-dshm\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.245324 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.245149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-home\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.245324 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.245263 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttgp2\" (UniqueName: \"kubernetes.io/projected/849becdb-894e-4545-8025-fa9b42b80845-kube-api-access-ttgp2\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.245444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.245323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-model-cache\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.245444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.245355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849becdb-894e-4545-8025-fa9b42b80845-tls-certs\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.245444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.245427 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-tmp-dir\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.346970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-model-cache\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347007 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.347013 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849becdb-894e-4545-8025-fa9b42b80845-tls-certs\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.347050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-tmp-dir\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.347110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-dshm\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.347170 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-home\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347254 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.347225 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttgp2\" (UniqueName: \"kubernetes.io/projected/849becdb-894e-4545-8025-fa9b42b80845-kube-api-access-ttgp2\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347516 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.347486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-model-cache\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347597 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.347519 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-tmp-dir\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.347597 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.347562 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-home\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.349515 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.349468 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-dshm\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.350074 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.350052 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849becdb-894e-4545-8025-fa9b42b80845-tls-certs\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.355722 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.355702 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttgp2\" (UniqueName: \"kubernetes.io/projected/849becdb-894e-4545-8025-fa9b42b80845-kube-api-access-ttgp2\") pod \"e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.412978 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.412943 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:20.692263 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:20.692113 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj"] Sep 17 18:44:20.695127 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:44:20.695098 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod849becdb_894e_4545_8025_fa9b42b80845.slice/crio-a017f632b429235f29292f8d4efb214128cdb51c913f694ca1e19178a76b3ff8 WatchSource:0}: Error finding container a017f632b429235f29292f8d4efb214128cdb51c913f694ca1e19178a76b3ff8: Status 404 returned error can't find the container with id a017f632b429235f29292f8d4efb214128cdb51c913f694ca1e19178a76b3ff8 Sep 17 18:44:21.424151 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:21.424115 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" event={"ID":"849becdb-894e-4545-8025-fa9b42b80845","Type":"ContainerStarted","Data":"d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad"} Sep 17 18:44:21.424537 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:21.424159 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" event={"ID":"849becdb-894e-4545-8025-fa9b42b80845","Type":"ContainerStarted","Data":"a017f632b429235f29292f8d4efb214128cdb51c913f694ca1e19178a76b3ff8"} Sep 17 18:44:21.444622 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:21.444573 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" podStartSLOduration=1.444558705 podStartE2EDuration="1.444558705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:44:21.442770073 +0000 UTC m=+2472.506714032" watchObservedRunningTime="2026-09-17 18:44:21.444558705 +0000 UTC m=+2472.508502662" Sep 17 18:44:30.413691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:30.413653 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:30.414274 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:30.413706 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:30.426226 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:30.426204 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:30.487786 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:30.487738 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:44:53.098390 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:53.098357 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj"] Sep 17 18:44:54.227794 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:44:54.227763 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj"] Sep 17 18:45:08.117728 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.117654 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" podUID="c79673c9-eaa5-499e-a4f6-4309ef5dfa02" containerName="main" containerID="cri-o://ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace" gracePeriod=45 Sep 17 18:45:08.372785 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.372728 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:45:08.423742 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.423707 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-home\") pod \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " Sep 17 18:45:08.423932 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.423763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tls-certs\") pod \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " Sep 17 18:45:08.423932 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.423807 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-model-cache\") pod \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " Sep 17 18:45:08.423932 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.423828 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tmp-dir\") pod \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " Sep 17 18:45:08.423932 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.423850 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-dshm\") pod \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " Sep 17 18:45:08.424168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.423987 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dklhm\" (UniqueName: \"kubernetes.io/projected/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-kube-api-access-dklhm\") pod \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\" (UID: \"c79673c9-eaa5-499e-a4f6-4309ef5dfa02\") " Sep 17 18:45:08.424168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.423996 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-home" (OuterVolumeSpecName: "home") pod "c79673c9-eaa5-499e-a4f6-4309ef5dfa02" (UID: "c79673c9-eaa5-499e-a4f6-4309ef5dfa02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:08.424168 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.424091 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-model-cache" (OuterVolumeSpecName: "model-cache") pod "c79673c9-eaa5-499e-a4f6-4309ef5dfa02" (UID: "c79673c9-eaa5-499e-a4f6-4309ef5dfa02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:08.424326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.424191 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c79673c9-eaa5-499e-a4f6-4309ef5dfa02" (UID: "c79673c9-eaa5-499e-a4f6-4309ef5dfa02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:08.424515 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.424495 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:08.424673 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.424519 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:08.424673 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.424532 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:08.426193 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.426175 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c79673c9-eaa5-499e-a4f6-4309ef5dfa02" (UID: "c79673c9-eaa5-499e-a4f6-4309ef5dfa02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:45:08.426479 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.426455 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-dshm" (OuterVolumeSpecName: "dshm") pod "c79673c9-eaa5-499e-a4f6-4309ef5dfa02" (UID: "c79673c9-eaa5-499e-a4f6-4309ef5dfa02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:08.426603 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.426585 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-kube-api-access-dklhm" (OuterVolumeSpecName: "kube-api-access-dklhm") pod "c79673c9-eaa5-499e-a4f6-4309ef5dfa02" (UID: "c79673c9-eaa5-499e-a4f6-4309ef5dfa02"). InnerVolumeSpecName "kube-api-access-dklhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:45:08.525066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.525038 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:08.525066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.525059 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:08.525066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.525069 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dklhm\" (UniqueName: \"kubernetes.io/projected/c79673c9-eaa5-499e-a4f6-4309ef5dfa02-kube-api-access-dklhm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:08.692963 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.692929 2582 generic.go:356] "Generic (PLEG): container finished" podID="c79673c9-eaa5-499e-a4f6-4309ef5dfa02" containerID="ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace" exitCode=0 Sep 17 18:45:08.693158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.692983 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" Sep 17 18:45:08.693158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.692994 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" event={"ID":"c79673c9-eaa5-499e-a4f6-4309ef5dfa02","Type":"ContainerDied","Data":"ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace"} Sep 17 18:45:08.693158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.693032 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj" event={"ID":"c79673c9-eaa5-499e-a4f6-4309ef5dfa02","Type":"ContainerDied","Data":"0ecd62e954d94743e90caf6fa9b4e49938d6a98d5874330cb838b7484487fc8e"} Sep 17 18:45:08.693158 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.693053 2582 scope.go:117] "RemoveContainer" containerID="ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace" Sep 17 18:45:08.702265 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.702249 2582 scope.go:117] "RemoveContainer" containerID="ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace" Sep 17 18:45:08.702547 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:45:08.702526 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace\": container with ID starting with ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace not found: ID does not exist" containerID="ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace" Sep 17 18:45:08.702615 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.702556 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace"} err="failed to get container status \"ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace\": rpc error: code = NotFound desc = could not find container \"ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace\": container with ID starting with ff846fec856e2d61ecadf9bbda39bc53915f97fa06e11435333fa2ef46042ace not found: ID does not exist" Sep 17 18:45:08.714547 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.714521 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj"] Sep 17 18:45:08.719123 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:08.719102 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-79ed8f9a-kserve-84f7996b99-gbbpj"] Sep 17 18:45:09.251551 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.251509 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" podUID="849becdb-894e-4545-8025-fa9b42b80845" containerName="main" containerID="cri-o://d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad" gracePeriod=45 Sep 17 18:45:09.346645 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.346611 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c79673c9-eaa5-499e-a4f6-4309ef5dfa02" path="/var/lib/kubelet/pods/c79673c9-eaa5-499e-a4f6-4309ef5dfa02/volumes" Sep 17 18:45:09.508743 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.508690 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:45:09.636247 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636214 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-model-cache\") pod \"849becdb-894e-4545-8025-fa9b42b80845\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " Sep 17 18:45:09.636430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636279 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttgp2\" (UniqueName: \"kubernetes.io/projected/849becdb-894e-4545-8025-fa9b42b80845-kube-api-access-ttgp2\") pod \"849becdb-894e-4545-8025-fa9b42b80845\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " Sep 17 18:45:09.636430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636326 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-dshm\") pod \"849becdb-894e-4545-8025-fa9b42b80845\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " Sep 17 18:45:09.636430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636355 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-home\") pod \"849becdb-894e-4545-8025-fa9b42b80845\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " Sep 17 18:45:09.636430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636376 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849becdb-894e-4545-8025-fa9b42b80845-tls-certs\") pod \"849becdb-894e-4545-8025-fa9b42b80845\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " Sep 17 18:45:09.636430 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636396 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-tmp-dir\") pod \"849becdb-894e-4545-8025-fa9b42b80845\" (UID: \"849becdb-894e-4545-8025-fa9b42b80845\") " Sep 17 18:45:09.636717 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636487 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-model-cache" (OuterVolumeSpecName: "model-cache") pod "849becdb-894e-4545-8025-fa9b42b80845" (UID: "849becdb-894e-4545-8025-fa9b42b80845"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:09.636717 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636639 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-home" (OuterVolumeSpecName: "home") pod "849becdb-894e-4545-8025-fa9b42b80845" (UID: "849becdb-894e-4545-8025-fa9b42b80845"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:09.636804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636731 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "849becdb-894e-4545-8025-fa9b42b80845" (UID: "849becdb-894e-4545-8025-fa9b42b80845"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:09.636804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636749 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-model-cache\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:09.636804 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.636770 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-home\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:09.638600 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.638570 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-dshm" (OuterVolumeSpecName: "dshm") pod "849becdb-894e-4545-8025-fa9b42b80845" (UID: "849becdb-894e-4545-8025-fa9b42b80845"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 18:45:09.638968 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.638948 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849becdb-894e-4545-8025-fa9b42b80845-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "849becdb-894e-4545-8025-fa9b42b80845" (UID: "849becdb-894e-4545-8025-fa9b42b80845"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:45:09.639037 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.639005 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849becdb-894e-4545-8025-fa9b42b80845-kube-api-access-ttgp2" (OuterVolumeSpecName: "kube-api-access-ttgp2") pod "849becdb-894e-4545-8025-fa9b42b80845" (UID: "849becdb-894e-4545-8025-fa9b42b80845"). InnerVolumeSpecName "kube-api-access-ttgp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:45:09.701072 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.701043 2582 generic.go:356] "Generic (PLEG): container finished" podID="849becdb-894e-4545-8025-fa9b42b80845" containerID="d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad" exitCode=0 Sep 17 18:45:09.701214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.701090 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" Sep 17 18:45:09.701214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.701106 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" event={"ID":"849becdb-894e-4545-8025-fa9b42b80845","Type":"ContainerDied","Data":"d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad"} Sep 17 18:45:09.701214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.701143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj" event={"ID":"849becdb-894e-4545-8025-fa9b42b80845","Type":"ContainerDied","Data":"a017f632b429235f29292f8d4efb214128cdb51c913f694ca1e19178a76b3ff8"} Sep 17 18:45:09.701214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.701165 2582 scope.go:117] "RemoveContainer" containerID="d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad" Sep 17 18:45:09.710719 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.710703 2582 scope.go:117] "RemoveContainer" containerID="d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad" Sep 17 18:45:09.711082 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:45:09.711060 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad\": container with ID starting with d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad not found: ID does not exist" containerID="d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad" Sep 17 18:45:09.711147 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.711089 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad"} err="failed to get container status \"d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad\": rpc error: code = NotFound desc = could not find container \"d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad\": container with ID starting with d0a04be7cc51a643a7bfcf98821101503e435e5c6ff6cf250b29fe277a1e27ad not found: ID does not exist" Sep 17 18:45:09.724390 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.724366 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj"] Sep 17 18:45:09.727858 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.727837 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-79ed8f9a-kserve-8f5f5594c-xzqmj"] Sep 17 18:45:09.738364 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.738338 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttgp2\" (UniqueName: \"kubernetes.io/projected/849becdb-894e-4545-8025-fa9b42b80845-kube-api-access-ttgp2\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:09.738438 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.738366 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-dshm\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:09.738438 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.738382 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849becdb-894e-4545-8025-fa9b42b80845-tls-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:09.738438 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:09.738395 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849becdb-894e-4545-8025-fa9b42b80845-tmp-dir\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:11.322547 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.322512 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn"] Sep 17 18:45:11.323228 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.323206 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="849becdb-894e-4545-8025-fa9b42b80845" containerName="main" Sep 17 18:45:11.323318 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.323233 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="849becdb-894e-4545-8025-fa9b42b80845" containerName="main" Sep 17 18:45:11.323318 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.323256 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79673c9-eaa5-499e-a4f6-4309ef5dfa02" containerName="main" Sep 17 18:45:11.323318 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.323264 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79673c9-eaa5-499e-a4f6-4309ef5dfa02" containerName="main" Sep 17 18:45:11.323495 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.323397 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="849becdb-894e-4545-8025-fa9b42b80845" containerName="main" Sep 17 18:45:11.323495 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.323417 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79673c9-eaa5-499e-a4f6-4309ef5dfa02" containerName="main" Sep 17 18:45:11.327904 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.327882 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.336212 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.336184 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn"] Sep 17 18:45:11.352714 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.352683 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="849becdb-894e-4545-8025-fa9b42b80845" path="/var/lib/kubelet/pods/849becdb-894e-4545-8025-fa9b42b80845/volumes" Sep 17 18:45:11.454798 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.454756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0804c04a-d1f4-4e19-b44a-5beed2db02d9-maas-api-tls\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.455015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.454806 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0804c04a-d1f4-4e19-b44a-5beed2db02d9-openshift-service-ca\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.455015 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.454954 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h9n2\" (UniqueName: \"kubernetes.io/projected/0804c04a-d1f4-4e19-b44a-5beed2db02d9-kube-api-access-8h9n2\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.455115 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.455013 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0804c04a-d1f4-4e19-b44a-5beed2db02d9-metrics-certs\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.556547 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.556500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0804c04a-d1f4-4e19-b44a-5beed2db02d9-metrics-certs\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.556987 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.556966 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0804c04a-d1f4-4e19-b44a-5beed2db02d9-maas-api-tls\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.557132 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.557118 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0804c04a-d1f4-4e19-b44a-5beed2db02d9-openshift-service-ca\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.558309 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.558285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h9n2\" (UniqueName: \"kubernetes.io/projected/0804c04a-d1f4-4e19-b44a-5beed2db02d9-kube-api-access-8h9n2\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.558590 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.558458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0804c04a-d1f4-4e19-b44a-5beed2db02d9-openshift-service-ca\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.560264 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.560212 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0804c04a-d1f4-4e19-b44a-5beed2db02d9-maas-api-tls\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.560444 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.560393 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0804c04a-d1f4-4e19-b44a-5beed2db02d9-metrics-certs\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.567591 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.567570 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h9n2\" (UniqueName: \"kubernetes.io/projected/0804c04a-d1f4-4e19-b44a-5beed2db02d9-kube-api-access-8h9n2\") pod \"maas-api-776dd778bc-v7xhn\" (UID: \"0804c04a-d1f4-4e19-b44a-5beed2db02d9\") " pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.642224 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.642137 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:11.925613 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:11.925580 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn"] Sep 17 18:45:11.929381 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:45:11.929355 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0804c04a_d1f4_4e19_b44a_5beed2db02d9.slice/crio-2409eaad42fe08bba04d3569c6f737f212ff67e619fb40470bd4b8f463c03659 WatchSource:0}: Error finding container 2409eaad42fe08bba04d3569c6f737f212ff67e619fb40470bd4b8f463c03659: Status 404 returned error can't find the container with id 2409eaad42fe08bba04d3569c6f737f212ff67e619fb40470bd4b8f463c03659 Sep 17 18:45:12.722224 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:12.722128 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" event={"ID":"0804c04a-d1f4-4e19-b44a-5beed2db02d9","Type":"ContainerStarted","Data":"250ad842f5a2810cd9b15b3e023734a05d78c7011723d73ff21a17d1d9ae474e"} Sep 17 18:45:12.722609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:12.722326 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:12.722609 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:12.722372 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" event={"ID":"0804c04a-d1f4-4e19-b44a-5beed2db02d9","Type":"ContainerStarted","Data":"2409eaad42fe08bba04d3569c6f737f212ff67e619fb40470bd4b8f463c03659"} Sep 17 18:45:12.740758 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:12.740705 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" podStartSLOduration=1.259057496 podStartE2EDuration="1.740684735s" totalImagesPullingTime="481.627239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:45:12.739409566 +0000 UTC m=+2523.803353527" watchObservedRunningTime="2026-09-17 18:45:12.740684735 +0000 UTC m=+2523.804628728" Sep 17 18:45:18.733965 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:18.733934 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-776dd778bc-v7xhn" Sep 17 18:45:26.876460 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:26.876425 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx"] Sep 17 18:45:26.876974 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:26.876689 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" podUID="9dca8211-671d-4466-a367-dcfc192a9718" containerName="maas-api" containerID="cri-o://8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572" gracePeriod=30 Sep 17 18:45:28.133791 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.133763 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:45:28.323376 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.323302 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-metrics-certs\") pod \"9dca8211-671d-4466-a367-dcfc192a9718\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " Sep 17 18:45:28.323519 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.323413 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dca8211-671d-4466-a367-dcfc192a9718-openshift-service-ca\") pod \"9dca8211-671d-4466-a367-dcfc192a9718\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " Sep 17 18:45:28.323519 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.323472 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9fxp\" (UniqueName: \"kubernetes.io/projected/9dca8211-671d-4466-a367-dcfc192a9718-kube-api-access-j9fxp\") pod \"9dca8211-671d-4466-a367-dcfc192a9718\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " Sep 17 18:45:28.323519 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.323504 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-maas-api-tls\") pod \"9dca8211-671d-4466-a367-dcfc192a9718\" (UID: \"9dca8211-671d-4466-a367-dcfc192a9718\") " Sep 17 18:45:28.323824 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.323792 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dca8211-671d-4466-a367-dcfc192a9718-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9dca8211-671d-4466-a367-dcfc192a9718" (UID: "9dca8211-671d-4466-a367-dcfc192a9718"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:45:28.325567 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.325539 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9dca8211-671d-4466-a367-dcfc192a9718" (UID: "9dca8211-671d-4466-a367-dcfc192a9718"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:45:28.325690 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.325590 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9dca8211-671d-4466-a367-dcfc192a9718" (UID: "9dca8211-671d-4466-a367-dcfc192a9718"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:45:28.325690 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.325587 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dca8211-671d-4466-a367-dcfc192a9718-kube-api-access-j9fxp" (OuterVolumeSpecName: "kube-api-access-j9fxp") pod "9dca8211-671d-4466-a367-dcfc192a9718" (UID: "9dca8211-671d-4466-a367-dcfc192a9718"). InnerVolumeSpecName "kube-api-access-j9fxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:45:28.425145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.425116 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fxp\" (UniqueName: \"kubernetes.io/projected/9dca8211-671d-4466-a367-dcfc192a9718-kube-api-access-j9fxp\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:28.425296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.425163 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-maas-api-tls\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:28.425296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.425172 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dca8211-671d-4466-a367-dcfc192a9718-metrics-certs\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:28.425296 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.425182 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dca8211-671d-4466-a367-dcfc192a9718-openshift-service-ca\") on node \"ip-10-0-137-104.ec2.internal\" DevicePath \"\"" Sep 17 18:45:28.807505 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.807470 2582 generic.go:356] "Generic (PLEG): container finished" podID="9dca8211-671d-4466-a367-dcfc192a9718" containerID="8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572" exitCode=0 Sep 17 18:45:28.807691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.807514 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" Sep 17 18:45:28.807691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.807543 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" event={"ID":"9dca8211-671d-4466-a367-dcfc192a9718","Type":"ContainerDied","Data":"8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572"} Sep 17 18:45:28.807691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.807585 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx" event={"ID":"9dca8211-671d-4466-a367-dcfc192a9718","Type":"ContainerDied","Data":"ee6db1d08e4959e2da6a191f2bb676832d904d7721d95c5c27ec77a0f55b398c"} Sep 17 18:45:28.807691 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.807606 2582 scope.go:117] "RemoveContainer" containerID="8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572" Sep 17 18:45:28.817746 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.817727 2582 scope.go:117] "RemoveContainer" containerID="8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572" Sep 17 18:45:28.818104 ip-10-0-137-104 kubenswrapper[2582]: E0917 18:45:28.818084 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572\": container with ID starting with 8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572 not found: ID does not exist" containerID="8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572" Sep 17 18:45:28.818165 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.818112 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572"} err="failed to get container status \"8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572\": rpc error: code = NotFound desc = could not find container \"8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572\": container with ID starting with 8426c47ca44e25bdcf8dcf94db76da21d0823bacef9ca43c0aab6d91c3215572 not found: ID does not exist" Sep 17 18:45:28.829968 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.829944 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx"] Sep 17 18:45:28.834145 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:28.834125 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5ffb5cf8dc-t4grx"] Sep 17 18:45:29.346658 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:29.346623 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dca8211-671d-4466-a367-dcfc192a9718" path="/var/lib/kubelet/pods/9dca8211-671d-4466-a367-dcfc192a9718/volumes" Sep 17 18:45:46.550114 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:46.550082 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bdzjt_247b1774-3590-4aa0-b528-ab6f38a22d7a/manager/0.log" Sep 17 18:45:46.681167 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:46.681135 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rftjv_083acc7b-dc74-462f-b658-3a69fc43b013/manager/0.log" Sep 17 18:45:46.815651 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:46.815557 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c8bff4649-k7g4p_846f3c5d-7aec-47b2-a72e-3006131207be/manager/0.log" Sep 17 18:45:47.321797 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:47.321770 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-6h5l5_c735fac7-125f-41fc-ab73-3a51824c554a/copy-manifests/0.log" Sep 17 18:45:47.340756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:47.340719 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-6h5l5_c735fac7-125f-41fc-ab73-3a51824c554a/manager/0.log" Sep 17 18:45:47.927503 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:47.927456 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-776dd778bc-v7xhn_0804c04a-d1f4-4e19-b44a-5beed2db02d9/maas-api/0.log" Sep 17 18:45:48.375770 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:48.375689 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wxss5_22fc0eb0-5968-4e5e-808d-b93e602ab9d6/postgres/0.log" Sep 17 18:45:49.259033 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:49.259002 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-m4n7n_2fed7e09-62e3-420e-bd69-2fc1ae276c41/manager/0.log" Sep 17 18:45:49.371414 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:49.371383 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-2pjz2_8b0de7a2-a141-408b-9385-9489b8903da2/manager/0.log" Sep 17 18:45:49.615116 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:49.615029 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-lq7l2_3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5/manager/0.log" Sep 17 18:45:49.861845 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:49.861819 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-g59qj_13af60a1-2f0c-4ff4-8393-83169bcabd09/manager/0.log" Sep 17 18:45:50.571375 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:50.571345 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bc8cbff6f-w2rcw_3626a029-7f14-49ac-a105-d51681cb58c2/kube-auth-proxy/0.log" Sep 17 18:45:50.698564 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:50.698538 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6v2jt_0f850692-e665-4012-bc51-ab3bd9c42ab0/istio-proxy/1.log" Sep 17 18:45:50.824162 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:50.824068 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kkjdk_edfdc12f-f3c4-426e-86fc-8cb583abc601/payload-pre-processing/0.log" Sep 17 18:45:50.950423 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:50.950384 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8k6rh_6927897e-3956-4585-8de2-bea9c1384dee/payload-processing/0.log" Sep 17 18:45:51.638524 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:51.638493 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-t5zjs_0bbaa877-0b3a-43a2-89a5-bafbf307e4f0/main/0.log" Sep 17 18:45:58.040383 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.040347 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vmxpt/must-gather-lrxt6"] Sep 17 18:45:58.040794 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.040777 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dca8211-671d-4466-a367-dcfc192a9718" containerName="maas-api" Sep 17 18:45:58.040794 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.040788 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dca8211-671d-4466-a367-dcfc192a9718" containerName="maas-api" Sep 17 18:45:58.040899 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.040873 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dca8211-671d-4466-a367-dcfc192a9718" containerName="maas-api" Sep 17 18:45:58.044437 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.044416 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vmxpt/must-gather-lrxt6" Sep 17 18:45:58.047292 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.047274 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vmxpt\"/\"kube-root-ca.crt\"" Sep 17 18:45:58.048262 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.048246 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vmxpt\"/\"openshift-service-ca.crt\"" Sep 17 18:45:58.048326 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.048250 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vmxpt\"/\"default-dockercfg-pgspx\"" Sep 17 18:45:58.055056 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.055034 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vmxpt/must-gather-lrxt6"] Sep 17 18:45:58.199139 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.199107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cch7\" (UniqueName: \"kubernetes.io/projected/e64a1193-f232-432b-af3f-508ee12ea755-kube-api-access-4cch7\") pod \"must-gather-lrxt6\" (UID: \"e64a1193-f232-432b-af3f-508ee12ea755\") " pod="openshift-must-gather-vmxpt/must-gather-lrxt6" Sep 17 18:45:58.199312 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.199177 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e64a1193-f232-432b-af3f-508ee12ea755-must-gather-output\") pod \"must-gather-lrxt6\" (UID: \"e64a1193-f232-432b-af3f-508ee12ea755\") " pod="openshift-must-gather-vmxpt/must-gather-lrxt6" Sep 17 18:45:58.300539 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.300458 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cch7\" (UniqueName: \"kubernetes.io/projected/e64a1193-f232-432b-af3f-508ee12ea755-kube-api-access-4cch7\") pod \"must-gather-lrxt6\" (UID: \"e64a1193-f232-432b-af3f-508ee12ea755\") " pod="openshift-must-gather-vmxpt/must-gather-lrxt6" Sep 17 18:45:58.300539 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.300511 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e64a1193-f232-432b-af3f-508ee12ea755-must-gather-output\") pod \"must-gather-lrxt6\" (UID: \"e64a1193-f232-432b-af3f-508ee12ea755\") " pod="openshift-must-gather-vmxpt/must-gather-lrxt6" Sep 17 18:45:58.300813 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.300794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e64a1193-f232-432b-af3f-508ee12ea755-must-gather-output\") pod \"must-gather-lrxt6\" (UID: \"e64a1193-f232-432b-af3f-508ee12ea755\") " pod="openshift-must-gather-vmxpt/must-gather-lrxt6" Sep 17 18:45:58.309455 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.309427 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cch7\" (UniqueName: \"kubernetes.io/projected/e64a1193-f232-432b-af3f-508ee12ea755-kube-api-access-4cch7\") pod \"must-gather-lrxt6\" (UID: \"e64a1193-f232-432b-af3f-508ee12ea755\") " pod="openshift-must-gather-vmxpt/must-gather-lrxt6" Sep 17 18:45:58.354894 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.354848 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vmxpt/must-gather-lrxt6" Sep 17 18:45:58.631227 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.631195 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vmxpt/must-gather-lrxt6"] Sep 17 18:45:58.633184 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:45:58.633155 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode64a1193_f232_432b_af3f_508ee12ea755.slice/crio-b05300b3c32b07ba631564446b3a0816c6f0764b2e39348bab2ecedc7a822b45 WatchSource:0}: Error finding container b05300b3c32b07ba631564446b3a0816c6f0764b2e39348bab2ecedc7a822b45: Status 404 returned error can't find the container with id b05300b3c32b07ba631564446b3a0816c6f0764b2e39348bab2ecedc7a822b45 Sep 17 18:45:58.635092 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.635074 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 18:45:58.978481 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:58.978446 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/must-gather-lrxt6" event={"ID":"e64a1193-f232-432b-af3f-508ee12ea755","Type":"ContainerStarted","Data":"b05300b3c32b07ba631564446b3a0816c6f0764b2e39348bab2ecedc7a822b45"} Sep 17 18:45:59.989576 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:59.989531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/must-gather-lrxt6" event={"ID":"e64a1193-f232-432b-af3f-508ee12ea755","Type":"ContainerStarted","Data":"a7acbcbd0a85be7b6dd7d76551913c33b2631112f823f45b8719a67f44cdc090"} Sep 17 18:45:59.990214 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:45:59.990168 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/must-gather-lrxt6" event={"ID":"e64a1193-f232-432b-af3f-508ee12ea755","Type":"ContainerStarted","Data":"e968bc733014ce305b475e5a701f85823f60d77bc75591ea09bd912a98d260e3"} Sep 17 18:46:00.009739 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:00.009679 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vmxpt/must-gather-lrxt6" podStartSLOduration=1.249277206 podStartE2EDuration="2.009659671s" totalImagesPullingTime="760.382465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:00.008337315 +0000 UTC m=+2571.072281272" watchObservedRunningTime="2026-09-17 18:46:00.009659671 +0000 UTC m=+2571.073603629" Sep 17 18:46:01.144627 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:01.144584 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tzwqq_912b1cfc-fb70-47aa-9558-f12d1d3f4416/global-pull-secret-syncer/0.log" Sep 17 18:46:01.280399 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:01.280361 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t4gc6_8c293e70-3fd6-4e79-86c7-123b5f7c7dee/konnectivity-agent/0.log" Sep 17 18:46:01.315978 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:01.315928 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-104.ec2.internal_a70d28bf90c68089cd4d8c2ce3d2b7fc/haproxy/0.log" Sep 17 18:46:06.236519 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:06.236480 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-m4n7n_2fed7e09-62e3-420e-bd69-2fc1ae276c41/manager/0.log" Sep 17 18:46:06.284395 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:06.284360 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-2pjz2_8b0de7a2-a141-408b-9385-9489b8903da2/manager/0.log" Sep 17 18:46:06.402557 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:06.402518 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-lq7l2_3c41efb9-0922-4c55-b7c0-8ec27cdcd8c5/manager/0.log" Sep 17 18:46:06.533940 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:06.533475 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-g59qj_13af60a1-2f0c-4ff4-8393-83169bcabd09/manager/0.log" Sep 17 18:46:07.930488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:07.930458 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1966b53-b777-4ad3-831e-0598bb3938e9/alertmanager/0.log" Sep 17 18:46:07.974317 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:07.974287 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1966b53-b777-4ad3-831e-0598bb3938e9/config-reloader/0.log" Sep 17 18:46:08.017829 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.017792 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1966b53-b777-4ad3-831e-0598bb3938e9/kube-rbac-proxy-web/0.log" Sep 17 18:46:08.073951 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.073918 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1966b53-b777-4ad3-831e-0598bb3938e9/kube-rbac-proxy/0.log" Sep 17 18:46:08.121985 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.121953 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1966b53-b777-4ad3-831e-0598bb3938e9/kube-rbac-proxy-metric/0.log" Sep 17 18:46:08.166756 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.166679 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1966b53-b777-4ad3-831e-0598bb3938e9/prom-label-proxy/0.log" Sep 17 18:46:08.218895 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.218763 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d1966b53-b777-4ad3-831e-0598bb3938e9/init-config-reloader/0.log" Sep 17 18:46:08.304224 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.303654 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-4xrmp_998bbc1b-3f82-4e86-93f1-0dce59233c20/cluster-monitoring-operator/0.log" Sep 17 18:46:08.357092 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.355947 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qh5nh_d4c07d5a-7b42-457b-b40d-5325d57e9500/kube-state-metrics/0.log" Sep 17 18:46:08.400637 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.400602 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qh5nh_d4c07d5a-7b42-457b-b40d-5325d57e9500/kube-rbac-proxy-main/0.log" Sep 17 18:46:08.450457 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.450430 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-qh5nh_d4c07d5a-7b42-457b-b40d-5325d57e9500/kube-rbac-proxy-self/0.log" Sep 17 18:46:08.503066 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.503016 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67bd5b5579-kmgxt_c3476733-5447-427c-8443-06f4319de249/metrics-server/0.log" Sep 17 18:46:08.791571 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.791485 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x8q6b_00fbb0f7-be24-4245-bac1-f346974d324e/node-exporter/0.log" Sep 17 18:46:08.831624 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.831593 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x8q6b_00fbb0f7-be24-4245-bac1-f346974d324e/kube-rbac-proxy/0.log" Sep 17 18:46:08.869323 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:08.869292 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-x8q6b_00fbb0f7-be24-4245-bac1-f346974d324e/init-textfile/0.log" Sep 17 18:46:09.023985 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:09.023955 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2774be6c-12e8-4b9a-a46f-edbda1d65655/prometheus/0.log" Sep 17 18:46:09.052603 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:09.052525 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2774be6c-12e8-4b9a-a46f-edbda1d65655/config-reloader/0.log" Sep 17 18:46:09.079603 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:09.079571 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2774be6c-12e8-4b9a-a46f-edbda1d65655/thanos-sidecar/0.log" Sep 17 18:46:09.104644 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:09.104612 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2774be6c-12e8-4b9a-a46f-edbda1d65655/kube-rbac-proxy-web/0.log" Sep 17 18:46:09.129788 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:09.129758 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2774be6c-12e8-4b9a-a46f-edbda1d65655/kube-rbac-proxy/0.log" Sep 17 18:46:09.158913 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:09.158880 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2774be6c-12e8-4b9a-a46f-edbda1d65655/kube-rbac-proxy-thanos/0.log" Sep 17 18:46:09.185302 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:09.185276 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2774be6c-12e8-4b9a-a46f-edbda1d65655/init-config-reloader/0.log" Sep 17 18:46:11.267956 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.267915 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4"] Sep 17 18:46:11.275340 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.275308 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.285562 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.285526 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4"] Sep 17 18:46:11.355146 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.355094 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-sys\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.355426 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.355406 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-lib-modules\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.355565 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.355548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-podres\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.355766 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.355751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4nkm\" (UniqueName: \"kubernetes.io/projected/0d5de245-309b-4915-8fc1-49c805f6e066-kube-api-access-r4nkm\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.355999 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.355982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-proc\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.457480 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.457430 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4nkm\" (UniqueName: \"kubernetes.io/projected/0d5de245-309b-4915-8fc1-49c805f6e066-kube-api-access-r4nkm\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.457676 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.457542 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-proc\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.457676 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.457613 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-sys\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.457676 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.457671 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-lib-modules\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.457940 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.457697 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-podres\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.457940 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.457877 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-podres\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.458260 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.458238 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-proc\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.458348 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.458301 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-sys\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.458759 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.458732 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d5de245-309b-4915-8fc1-49c805f6e066-lib-modules\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.468283 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.468255 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4nkm\" (UniqueName: \"kubernetes.io/projected/0d5de245-309b-4915-8fc1-49c805f6e066-kube-api-access-r4nkm\") pod \"perf-node-gather-daemonset-27dw4\" (UID: \"0d5de245-309b-4915-8fc1-49c805f6e066\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.518195 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.518111 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/1.log" Sep 17 18:46:11.527290 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.527187 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-4fqtn_576187ca-ff17-474c-b9e5-c8b17ac8e982/console-operator/2.log" Sep 17 18:46:11.592488 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.592452 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:11.903755 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:11.903727 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4"] Sep 17 18:46:11.907439 ip-10-0-137-104 kubenswrapper[2582]: W0917 18:46:11.907404 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d5de245_309b_4915_8fc1_49c805f6e066.slice/crio-a127579e773db1f932b68d56039a62901784e210c82378466d557916fb06b5e4 WatchSource:0}: Error finding container a127579e773db1f932b68d56039a62901784e210c82378466d557916fb06b5e4: Status 404 returned error can't find the container with id a127579e773db1f932b68d56039a62901784e210c82378466d557916fb06b5e4 Sep 17 18:46:12.103898 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:12.103839 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" event={"ID":"0d5de245-309b-4915-8fc1-49c805f6e066","Type":"ContainerStarted","Data":"f52752e832811d9c9155c1561fc69e5ff6395408cd222b9f1b2a764ff55ef145"} Sep 17 18:46:12.104095 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:12.103923 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" event={"ID":"0d5de245-309b-4915-8fc1-49c805f6e066","Type":"ContainerStarted","Data":"a127579e773db1f932b68d56039a62901784e210c82378466d557916fb06b5e4"} Sep 17 18:46:12.104169 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:12.104138 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:12.122748 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:12.122694 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" podStartSLOduration=1.122675899 podStartE2EDuration="1.122675899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:12.120742704 +0000 UTC m=+2583.184686663" watchObservedRunningTime="2026-09-17 18:46:12.122675899 +0000 UTC m=+2583.186619856" Sep 17 18:46:12.140136 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:12.140104 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cfd75d48b-cbnf6_13865664-45f9-45eb-a30c-341c56a07e50/console/0.log" Sep 17 18:46:12.184093 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:12.184043 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-kfj8l_14548da0-26ac-4c8e-b0d9-a5da4aa48090/download-server/0.log" Sep 17 18:46:12.904892 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:12.903425 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-bj4gx_1c794bd0-5e63-4cb0-aaef-8c909d75ea1a/volume-data-source-validator/0.log" Sep 17 18:46:14.173297 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:14.173260 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-42kg8_aeee9793-7091-4047-b99c-871e17adafa0/dns/0.log" Sep 17 18:46:14.211579 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:14.211549 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-42kg8_aeee9793-7091-4047-b99c-871e17adafa0/kube-rbac-proxy/0.log" Sep 17 18:46:14.405340 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:14.405310 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-spg6t_23d3772b-6060-4903-be8a-49a0704c9a3e/dns-node-resolver/0.log" Sep 17 18:46:15.108617 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:15.108577 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ktpwl_7398a155-5335-448b-b8b5-c86efaf5e6b6/node-ca/0.log" Sep 17 18:46:16.629059 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:16.629021 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bc8cbff6f-w2rcw_3626a029-7f14-49ac-a105-d51681cb58c2/kube-auth-proxy/0.log" Sep 17 18:46:16.660010 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:16.659965 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6v2jt_0f850692-e665-4012-bc51-ab3bd9c42ab0/istio-proxy/0.log" Sep 17 18:46:16.666153 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:16.666125 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6v2jt_0f850692-e665-4012-bc51-ab3bd9c42ab0/istio-proxy/1.log" Sep 17 18:46:16.710360 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:16.710321 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kkjdk_edfdc12f-f3c4-426e-86fc-8cb583abc601/payload-pre-processing/0.log" Sep 17 18:46:16.761650 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:16.761621 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8k6rh_6927897e-3956-4585-8de2-bea9c1384dee/payload-processing/0.log" Sep 17 18:46:17.740487 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:17.740455 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4tllr_bfa0606a-1289-4a5d-8271-f17430d5e740/serve-healthcheck-canary/0.log" Sep 17 18:46:18.123589 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:18.123503 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-27dw4" Sep 17 18:46:18.400586 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:18.400562 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xfxv_e8342fcb-842b-4f54-8f5b-7d34215c9091/kube-rbac-proxy/0.log" Sep 17 18:46:18.424415 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:18.424385 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xfxv_e8342fcb-842b-4f54-8f5b-7d34215c9091/exporter/0.log" Sep 17 18:46:18.449042 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:18.449014 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xfxv_e8342fcb-842b-4f54-8f5b-7d34215c9091/extractor/0.log" Sep 17 18:46:20.922100 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:20.922064 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bdzjt_247b1774-3590-4aa0-b528-ab6f38a22d7a/manager/0.log" Sep 17 18:46:21.467220 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:21.467192 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rftjv_083acc7b-dc74-462f-b658-3a69fc43b013/manager/0.log" Sep 17 18:46:21.558198 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:21.558156 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7c8bff4649-k7g4p_846f3c5d-7aec-47b2-a72e-3006131207be/manager/0.log" Sep 17 18:46:21.864317 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:21.864264 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-6h5l5_c735fac7-125f-41fc-ab73-3a51824c554a/manager/0.log" Sep 17 18:46:21.884924 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:21.884890 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-6h5l5_c735fac7-125f-41fc-ab73-3a51824c554a/copy-manifests/0.log" Sep 17 18:46:22.826034 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:22.826008 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bc8cbff6f-w2rcw_3626a029-7f14-49ac-a105-d51681cb58c2/kube-auth-proxy/0.log" Sep 17 18:46:22.848718 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:22.848684 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6v2jt_0f850692-e665-4012-bc51-ab3bd9c42ab0/istio-proxy/0.log" Sep 17 18:46:22.852143 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:22.852107 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6v2jt_0f850692-e665-4012-bc51-ab3bd9c42ab0/istio-proxy/1.log" Sep 17 18:46:22.885166 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:22.885138 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kkjdk_edfdc12f-f3c4-426e-86fc-8cb583abc601/payload-pre-processing/0.log" Sep 17 18:46:22.928755 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:22.928695 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8k6rh_6927897e-3956-4585-8de2-bea9c1384dee/payload-processing/0.log" Sep 17 18:46:23.311955 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:23.311923 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tfmcf_0b903663-c19e-409a-ad56-44268c75af1e/openshift-lws-operator/0.log" Sep 17 18:46:24.897216 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:24.897182 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bc8cbff6f-w2rcw_3626a029-7f14-49ac-a105-d51681cb58c2/kube-auth-proxy/0.log" Sep 17 18:46:24.920846 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:24.920815 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6v2jt_0f850692-e665-4012-bc51-ab3bd9c42ab0/istio-proxy/0.log" Sep 17 18:46:24.926579 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:24.926550 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-6v2jt_0f850692-e665-4012-bc51-ab3bd9c42ab0/istio-proxy/1.log" Sep 17 18:46:24.966009 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:24.965978 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kkjdk_edfdc12f-f3c4-426e-86fc-8cb583abc601/payload-pre-processing/0.log" Sep 17 18:46:25.012307 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:25.012275 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8k6rh_6927897e-3956-4585-8de2-bea9c1384dee/payload-processing/0.log" Sep 17 18:46:28.922352 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:28.922319 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-frch6_661c1262-8856-4aa9-be54-f3a2594cee26/kube-storage-version-migrator-operator/1.log" Sep 17 18:46:28.924265 ip-10-0-137-104 kubenswrapper[2582]: I0917 18:46:28.924241 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-frch6_661c1262-8856-4aa9-be54-f3a2594cee26/kube-storage-version-migrator-operator/0.log"