Sep 15 11:38:30.842659 ip-10-0-140-16 systemd[1]: Starting Kubernetes Kubelet... Sep 15 11:38:31.272734 ip-10-0-140-16 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 11:38:31.272734 ip-10-0-140-16 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 15 11:38:31.272734 ip-10-0-140-16 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 11:38:31.272734 ip-10-0-140-16 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 15 11:38:31.272734 ip-10-0-140-16 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 15 11:38:31.273463 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.272823 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 15 11:38:31.279098 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279070 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:38:31.279098 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279095 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:38:31.279098 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279099 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:38:31.279098 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279102 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279105 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279108 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279111 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279114 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279116 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279118 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279122 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279125 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279128 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279130 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279133 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279136 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279138 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279149 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279151 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279155 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279157 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279160 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279163 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:38:31.279223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279166 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279168 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279170 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279173 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279175 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279178 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279180 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279183 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279185 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279188 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279190 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279193 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279195 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279198 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279201 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279204 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279207 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279209 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279212 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279215 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:38:31.279726 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279217 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279220 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279223 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279226 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279228 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279231 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279234 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279237 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279239 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279243 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279245 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279248 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279250 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279253 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279255 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279257 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279260 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279262 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279266 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279268 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:38:31.280248 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279271 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279276 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279280 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279283 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279301 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279304 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279308 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279311 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279314 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279317 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279321 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279324 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279328 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279331 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279334 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279336 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279339 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279342 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279346 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:38:31.280799 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279349 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279352 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279355 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279358 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279361 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279364 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279366 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279369 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279371 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279376 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279380 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279382 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279385 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279387 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279390 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279392 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.279395 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280239 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280248 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280252 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:38:31.281264 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280255 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280258 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280261 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280264 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280268 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280270 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280273 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280276 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280278 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280281 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280284 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280431 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280436 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280439 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280442 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280445 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280448 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280451 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280453 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:38:31.281803 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280457 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280462 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280465 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280468 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280471 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280474 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280477 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280480 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280483 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280486 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280488 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280492 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280495 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280498 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280501 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280504 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280507 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280510 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280513 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280516 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:38:31.282418 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280518 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280521 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280524 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280527 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280531 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280533 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280536 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280539 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280541 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280544 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280546 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280550 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280552 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280556 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280560 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280563 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280565 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280568 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280572 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:38:31.283185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280575 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280578 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280581 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280584 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280587 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280590 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280592 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280595 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280597 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280600 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280603 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280605 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280608 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280610 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280615 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280618 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280620 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280623 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280626 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280628 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:38:31.283773 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280631 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280633 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280636 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280638 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280641 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280643 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280646 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280649 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280651 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280653 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280656 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280659 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280661 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280664 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280666 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280669 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280673 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.280675 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281091 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281096 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:38:31.284272 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281099 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281101 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281105 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281108 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281111 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281114 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281116 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281119 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281121 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281124 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281134 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281137 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281140 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281143 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281145 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281148 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281151 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281153 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281157 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:38:31.284808 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281161 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281165 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281167 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281171 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281174 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281177 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281180 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281182 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281185 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281188 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281190 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281193 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281195 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281198 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281201 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281203 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281206 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281210 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281213 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:38:31.285303 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281216 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281218 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281221 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281224 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281226 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281237 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281240 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281242 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281245 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281247 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281250 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281252 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281255 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281258 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281261 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281264 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281266 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281268 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281271 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281274 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:38:31.285786 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281276 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281279 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281282 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281306 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281311 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281315 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281318 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281322 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281326 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281329 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281334 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281337 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281340 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281342 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281345 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281347 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281350 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281352 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281365 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281368 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:38:31.286331 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281371 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281373 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281376 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281378 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281383 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281385 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281388 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281391 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281394 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281396 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281399 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281401 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281404 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281406 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281409 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281411 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281414 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281416 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.281419 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281506 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281540 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 15 11:38:31.286830 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281547 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281552 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281556 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281559 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281564 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281568 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281571 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281574 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281577 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281581 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281590 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281593 2579 flags.go:64] FLAG: --cgroup-root="" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281596 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281599 2579 flags.go:64] FLAG: --client-ca-file="" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281602 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281606 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281616 2579 flags.go:64] FLAG: --cluster-domain="" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281618 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281622 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281625 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281628 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281633 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281636 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 15 11:38:31.287353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281639 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281643 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281646 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281649 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281651 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281654 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281657 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281661 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281664 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281667 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281676 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281679 2579 flags.go:64] FLAG: --enable-server="true" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281682 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281689 2579 flags.go:64] FLAG: --event-burst="100" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281692 2579 flags.go:64] FLAG: --event-qps="50" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281694 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281697 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281700 2579 flags.go:64] FLAG: --eviction-hard="" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281704 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281707 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281715 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281719 2579 flags.go:64] FLAG: --eviction-soft="" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281721 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281725 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281729 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 15 11:38:31.287918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281732 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281734 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281737 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281740 2579 flags.go:64] FLAG: --feature-gates="" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281744 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281747 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281750 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281753 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281756 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281759 2579 flags.go:64] FLAG: --help="false" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281762 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281765 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281768 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281771 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281774 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281777 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281780 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281783 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281786 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281789 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281791 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281795 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281797 2579 flags.go:64] FLAG: --kube-reserved="" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281800 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 15 11:38:31.288704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281803 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281806 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281809 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281812 2579 flags.go:64] FLAG: --lock-file="" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281821 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281824 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281827 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281833 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281836 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281839 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281842 2579 flags.go:64] FLAG: --logging-format="text" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281844 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281848 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281850 2579 flags.go:64] FLAG: --manifest-url="" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281853 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281858 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281861 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281865 2579 flags.go:64] FLAG: --max-pods="110" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281868 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281871 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281874 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281876 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281879 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281882 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281885 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 15 11:38:31.289327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281893 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281896 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281898 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281901 2579 flags.go:64] FLAG: --pod-cidr="" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281905 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281915 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281918 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281921 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281925 2579 flags.go:64] FLAG: --port="10250" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281928 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281931 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06a175787eaa41272" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281934 2579 flags.go:64] FLAG: --qos-reserved="" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281943 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281946 2579 flags.go:64] FLAG: --register-node="true" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281949 2579 flags.go:64] FLAG: --register-with-taints="" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281953 2579 flags.go:64] FLAG: --registry-burst="10" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281955 2579 flags.go:64] FLAG: --registry-qps="5" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281958 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281960 2579 flags.go:64] FLAG: --reserved-memory="" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281964 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281967 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281970 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281973 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281975 2579 flags.go:64] FLAG: --runonce="false" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281978 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 15 11:38:31.289984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281981 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281984 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281987 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281989 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281992 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281995 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.281998 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282001 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282004 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282007 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282009 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282012 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282015 2579 flags.go:64] FLAG: --system-cgroups="" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282018 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282023 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282027 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282030 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282398 2579 flags.go:64] FLAG: --tls-min-version="" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282405 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282412 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282520 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282527 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282534 2579 flags.go:64] FLAG: --v="2" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282541 2579 flags.go:64] FLAG: --version="false" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282548 2579 flags.go:64] FLAG: --vmodule="" Sep 15 11:38:31.290720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282560 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.282566 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283066 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283075 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283080 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283085 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283090 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283094 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283098 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283102 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283108 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283114 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283119 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283129 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283134 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283138 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283142 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283146 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283150 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283154 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:38:31.291821 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283159 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283163 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283170 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283174 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283178 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283183 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283191 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283196 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283231 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283236 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283240 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283245 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283248 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283252 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283257 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283261 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283265 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283268 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283278 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283282 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:38:31.292920 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283303 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283307 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283312 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283316 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283320 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283324 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283328 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283332 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283336 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283340 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283347 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283356 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283360 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283364 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283369 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283373 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283378 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283381 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283385 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283389 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:38:31.293705 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283393 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283470 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283499 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283509 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283554 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283659 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283665 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283669 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283672 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283675 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283678 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283680 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283683 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283686 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283688 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283691 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283693 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283696 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283698 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283701 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:38:31.294223 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283705 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283707 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283710 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283713 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283716 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283720 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283725 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283728 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283731 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283734 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283737 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283739 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283744 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283747 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283750 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283753 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283756 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283759 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283762 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:38:31.294985 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283765 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.283767 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.284527 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.293487 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.293515 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.294475 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.294491 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294591 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294600 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294607 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294612 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294616 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294621 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294626 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:38:31.295961 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294630 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294634 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294639 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294646 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294652 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294657 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294662 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294666 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294671 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294675 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294681 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294685 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294689 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294694 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294699 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294703 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294707 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294711 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294716 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:38:31.296414 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294723 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294729 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294734 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294739 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294744 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294749 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294753 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294758 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294762 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294766 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294770 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294775 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294779 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294783 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294788 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294792 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294796 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294800 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294804 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:38:31.297053 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294808 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294813 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294817 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294821 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294825 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294829 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294833 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294837 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294841 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294845 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294849 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294853 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294858 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294862 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294866 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294873 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294880 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294885 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294889 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294893 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:38:31.297838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294898 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294902 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294905 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294910 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294914 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294918 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294922 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294927 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294932 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294936 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294941 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294948 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294952 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294957 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294961 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294966 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294971 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294975 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294979 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294983 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:38:31.298592 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294987 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294991 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294994 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.294998 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295002 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295006 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295010 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295014 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295019 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295024 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295028 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295033 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295038 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295042 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.295050 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 11:38:31.299218 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295251 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295261 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295265 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295270 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295275 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295281 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295310 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295315 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295320 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295325 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295329 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295333 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295337 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295342 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295347 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295350 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295354 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295359 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 15 11:38:31.299658 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295364 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295369 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295374 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295378 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295384 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295388 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295392 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295397 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295402 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295407 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295411 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295416 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295420 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295424 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295428 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295433 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295437 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295441 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295446 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295450 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 15 11:38:31.300147 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295454 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295459 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295463 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295467 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295472 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295477 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295481 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295485 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295489 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295493 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295498 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295502 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295506 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295510 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295515 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295519 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295524 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295529 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295533 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295538 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 15 11:38:31.300697 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295542 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295547 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295552 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295556 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295560 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295564 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295568 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295572 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295576 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295580 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295585 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295590 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295594 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295598 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295603 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295607 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295611 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295617 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295622 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295626 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295630 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 15 11:38:31.301260 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295634 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295638 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295642 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295647 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295651 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295655 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295659 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295664 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295668 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295673 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295678 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295682 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295687 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295692 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295698 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295702 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295707 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295711 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295715 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 15 11:38:31.301881 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:31.295719 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 15 11:38:31.302427 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.295728 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 15 11:38:31.302427 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.295931 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 15 11:38:31.302427 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.299033 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 15 11:38:31.302427 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.300106 2579 server.go:1019] "Starting client certificate rotation" Sep 15 11:38:31.302427 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.300217 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 11:38:31.302427 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.300935 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 15 11:38:31.303107 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.303085 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 11:38:31.304561 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.304528 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 15 11:38:31.318690 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.318668 2579 log.go:25] "Validated CRI v1 runtime API" Sep 15 11:38:31.323156 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.323134 2579 log.go:25] "Validated CRI v1 image API" Sep 15 11:38:31.324953 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.324924 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 15 11:38:31.329376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.329348 2579 fs.go:135] Filesystem UUIDs: map[02eebfda-37d1-4c3f-b585-33b9ff30c66d:/dev/nvme0n1p4 5756fad6-2368-4e50-976a-f6b751a4f6d7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 15 11:38:31.329376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.329369 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 15 11:38:31.335596 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.335476 2579 manager.go:216] Machine: {Timestamp:2026-09-15 11:38:31.334146367 +0000 UTC m=+0.379166513 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3078666 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e4ec7e4be703f37e341c4b58fa53d SystemUUID:ec2e4ec7-e4be-703f-37e3-41c4b58fa53d BootID:f1afa587-f0e2-440a-ab3f-19e8a09ec1a6 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:d1:10:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:d1:10:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:96:4b:8d:32:72 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 15 11:38:31.335596 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.335589 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 15 11:38:31.335738 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.335725 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 15 11:38:31.339163 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.339130 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 15 11:38:31.339331 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.339167 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 15 11:38:31.340132 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.340121 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 15 11:38:31.340166 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.340134 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 15 11:38:31.340166 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.340142 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 11:38:31.340820 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.340809 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 15 11:38:31.341834 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.341823 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 15 11:38:31.341897 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.341881 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 11:38:31.347041 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347025 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 15 11:38:31.347171 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347161 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 15 11:38:31.347202 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347193 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 15 11:38:31.347241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347204 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 15 11:38:31.347241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347223 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 15 11:38:31.347241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347234 2579 kubelet.go:398] "Adding apiserver pod source" Sep 15 11:38:31.347329 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347244 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 15 11:38:31.347754 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347743 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 11:38:31.347789 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.347771 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 15 11:38:31.350618 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.350595 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 15 11:38:31.354810 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.354790 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 15 11:38:31.354900 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.354826 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 15 11:38:31.356872 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.356845 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 15 11:38:31.356930 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.356845 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 15 11:38:31.356989 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.356977 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 15 11:38:31.357024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.356995 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 15 11:38:31.357024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357001 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 15 11:38:31.357024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357007 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 15 11:38:31.357024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357016 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 15 11:38:31.357024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357021 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 15 11:38:31.357153 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357035 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 15 11:38:31.357153 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357040 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 15 11:38:31.357153 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357047 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 15 11:38:31.357153 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357053 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 15 11:38:31.357153 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357062 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 15 11:38:31.357153 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.357069 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 15 11:38:31.359150 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.359139 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 15 11:38:31.359150 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.359151 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 15 11:38:31.363026 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.363010 2579 server.go:1268] "Started kubelet" Sep 15 11:38:31.363199 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.363133 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 15 11:38:31.363314 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.363215 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 15 11:38:31.363314 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.363169 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 15 11:38:31.364006 ip-10-0-140-16 systemd[1]: Started Kubernetes Kubelet. Sep 15 11:38:31.364511 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.364478 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 15 11:38:31.369599 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.369574 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 15 11:38:31.370911 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.370873 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 15 11:38:31.371822 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.371798 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 15 11:38:31.372924 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.371877 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-16.ec2.internal.18d57b960e2ff3de default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-16.ec2.internal,UID:ip-10-0-140-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-16.ec2.internal,},FirstTimestamp:2026-09-15 11:38:31.362974686 +0000 UTC m=+0.407994830,LastTimestamp:2026-09-15 11:38:31.362974686 +0000 UTC m=+0.407994830,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-16.ec2.internal,}" Sep 15 11:38:31.374585 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.374566 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dcx87" Sep 15 11:38:31.376044 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.376004 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 15 11:38:31.376655 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.376628 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 15 11:38:31.376746 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.376722 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:31.377546 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377530 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 15 11:38:31.377723 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377709 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 15 11:38:31.377723 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377725 2579 factory.go:55] Registering systemd factory Sep 15 11:38:31.377821 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377731 2579 factory.go:223] Registration of the systemd container factory successfully Sep 15 11:38:31.377821 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377785 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 15 11:38:31.377821 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377795 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 15 11:38:31.377982 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377970 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 15 11:38:31.378034 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377983 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 15 11:38:31.378034 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.377972 2579 factory.go:153] Registering CRI-O factory Sep 15 11:38:31.378034 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.378007 2579 factory.go:223] Registration of the crio container factory successfully Sep 15 11:38:31.378034 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.378034 2579 factory.go:103] Registering Raw factory Sep 15 11:38:31.378195 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.378047 2579 manager.go:1196] Started watching for new ooms in manager Sep 15 11:38:31.378449 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.378437 2579 manager.go:349] Starting recovery of all containers Sep 15 11:38:31.378584 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.378561 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 15 11:38:31.378641 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.378583 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 15 11:38:31.380941 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.380920 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dcx87" Sep 15 11:38:31.388018 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.387996 2579 manager.go:354] Recovery completed Sep 15 11:38:31.393437 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.393424 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:38:31.395911 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.395893 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:38:31.395986 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.395929 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:38:31.395986 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.395942 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:38:31.396625 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.396612 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 15 11:38:31.396625 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.396625 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 15 11:38:31.396761 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.396645 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 15 11:38:31.396761 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.396650 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:38:31.397405 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.397391 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:38:31.397481 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.397422 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:38:31.397481 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.397438 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:38:31.398432 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.398418 2579 policy_none.go:49] "None policy: Start" Sep 15 11:38:31.398482 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.398437 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 15 11:38:31.398482 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.398448 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 15 11:38:31.400198 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.400186 2579 policy_none.go:47] "Start" Sep 15 11:38:31.443001 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.442963 2579 manager.go:337] "Starting Device Plugin manager" Sep 15 11:38:31.443001 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.443005 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 15 11:38:31.461384 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.443016 2579 server.go:87] "Starting device plugin registration server" Sep 15 11:38:31.461384 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.443340 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 15 11:38:31.461384 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.443355 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 15 11:38:31.461384 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.443433 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 15 11:38:31.461384 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.443520 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 15 11:38:31.461384 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.443529 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 15 11:38:31.461384 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.444130 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 15 11:38:31.461384 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.444184 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:31.480049 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.479999 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 15 11:38:31.481370 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.481345 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 15 11:38:31.481465 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.481404 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 15 11:38:31.481465 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.481432 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 15 11:38:31.481563 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.481480 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 15 11:38:31.483577 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.483559 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:38:31.543611 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.543497 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:38:31.544595 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.544575 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:38:31.544687 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.544609 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:38:31.544687 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.544624 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:38:31.544687 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.544648 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.550933 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.550918 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.550980 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.550941 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-16.ec2.internal\": node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:31.564227 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.564198 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:31.581755 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.581734 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal"] Sep 15 11:38:31.581839 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.581793 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:38:31.583376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.583358 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:38:31.583457 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.583389 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:38:31.583457 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.583401 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:38:31.585454 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.585442 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:38:31.585571 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.585553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.585622 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.585585 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:38:31.586238 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.586224 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:38:31.586332 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.586245 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:38:31.586332 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.586259 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:38:31.586332 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.586301 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:38:31.586332 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.586327 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:38:31.586460 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.586346 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:38:31.588334 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.588319 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.588423 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.588353 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 15 11:38:31.589067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.589052 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientMemory" Sep 15 11:38:31.589147 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.589078 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasNoDiskPressure" Sep 15 11:38:31.589147 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.589087 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeHasSufficientPID" Sep 15 11:38:31.613884 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.613865 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-16.ec2.internal\" not found" node="ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.617219 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.617205 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-16.ec2.internal\" not found" node="ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.665093 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.665074 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:31.679484 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.679459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d02dff20630896118b374c2cd0a78b5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-16.ec2.internal\" (UID: \"2d02dff20630896118b374c2cd0a78b5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.679568 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.679488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8319835104d09724226f87792cc29979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal\" (UID: \"8319835104d09724226f87792cc29979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.679568 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.679506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8319835104d09724226f87792cc29979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal\" (UID: \"8319835104d09724226f87792cc29979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.765912 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.765869 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:31.780344 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.780323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8319835104d09724226f87792cc29979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal\" (UID: \"8319835104d09724226f87792cc29979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.780431 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.780353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d02dff20630896118b374c2cd0a78b5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-16.ec2.internal\" (UID: \"2d02dff20630896118b374c2cd0a78b5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.780431 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.780372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8319835104d09724226f87792cc29979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal\" (UID: \"8319835104d09724226f87792cc29979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.780494 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.780432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2d02dff20630896118b374c2cd0a78b5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-16.ec2.internal\" (UID: \"2d02dff20630896118b374c2cd0a78b5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.780494 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.780438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8319835104d09724226f87792cc29979-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal\" (UID: \"8319835104d09724226f87792cc29979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.780558 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.780495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8319835104d09724226f87792cc29979-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal\" (UID: \"8319835104d09724226f87792cc29979\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.866884 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.866785 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:31.916432 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.916396 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.919319 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:31.919275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:31.967799 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:31.967759 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:32.068366 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:32.068332 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:32.169010 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:32.168923 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:32.269882 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:32.269842 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:32.271198 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.271183 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:38:32.299636 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.299599 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 15 11:38:32.300085 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.299768 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 11:38:32.300085 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.299777 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 15 11:38:32.370094 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:32.370062 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:32.384904 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.384849 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 11:33:33 +0000 UTC" deadline="2028-06-11 04:00:25.50835904 +0000 UTC" Sep 15 11:38:32.384904 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.384897 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15232h21m53.123466828s" Sep 15 11:38:32.471346 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:32.471120 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-16.ec2.internal\" not found" Sep 15 11:38:32.489219 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.489185 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:38:32.541368 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:32.541334 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8319835104d09724226f87792cc29979.slice/crio-cb81a96aebd57eb8acc96d0a25e2ea817f049cf8bf231346f6b8efc28b1fcc6e WatchSource:0}: Error finding container cb81a96aebd57eb8acc96d0a25e2ea817f049cf8bf231346f6b8efc28b1fcc6e: Status 404 returned error can't find the container with id cb81a96aebd57eb8acc96d0a25e2ea817f049cf8bf231346f6b8efc28b1fcc6e Sep 15 11:38:32.541588 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:32.541568 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d02dff20630896118b374c2cd0a78b5.slice/crio-23245325250b393fa513053aa9ae2fe39f9c779d29dec82171ec3eebf74489b2 WatchSource:0}: Error finding container 23245325250b393fa513053aa9ae2fe39f9c779d29dec82171ec3eebf74489b2: Status 404 returned error can't find the container with id 23245325250b393fa513053aa9ae2fe39f9c779d29dec82171ec3eebf74489b2 Sep 15 11:38:32.545909 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.545890 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 11:38:32.577652 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.577618 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" Sep 15 11:38:32.588905 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.588879 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 11:38:32.590703 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.590687 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" Sep 15 11:38:32.600518 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.600494 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 15 11:38:32.735280 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:32.735184 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:38:35.048406 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.048361 2579 apiserver.go:52] "Watching apiserver" Sep 15 11:38:35.054624 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.054594 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 15 11:38:35.056333 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.055877 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-p469s","kube-system/konnectivity-agent-k5k2z","kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4","openshift-dns/node-resolver-2kk9v","openshift-image-registry/node-ca-69tt7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal","openshift-multus/multus-qrf5k","openshift-cluster-node-tuning-operator/tuned-5bzbl","openshift-multus/multus-additional-cni-plugins-7j5wd","openshift-multus/network-metrics-daemon-q7vl8","openshift-network-diagnostics/network-check-target-4ztt5","openshift-network-operator/iptables-alerter-nvknt","openshift-ovn-kubernetes/ovnkube-node-jhwc4"] Sep 15 11:38:35.059384 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.058695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.060908 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.060891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.061365 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.061332 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 15 11:38:35.061464 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.061343 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 15 11:38:35.061550 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.061530 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 15 11:38:35.061898 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.061832 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qdc4d\"" Sep 15 11:38:35.061898 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.061859 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 15 11:38:35.062067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.061971 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 15 11:38:35.062973 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.062940 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 15 11:38:35.062973 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.062968 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:38:35.063387 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.063371 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 15 11:38:35.063579 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.063560 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tlgxd\"" Sep 15 11:38:35.063933 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.063918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.065979 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.065959 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 15 11:38:35.066229 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.066180 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 15 11:38:35.066310 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.066239 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-844l7\"" Sep 15 11:38:35.066519 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.066504 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 15 11:38:35.066874 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.066856 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 15 11:38:35.067093 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.067078 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 15 11:38:35.067164 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.067152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.067245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.067078 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 15 11:38:35.069128 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.069108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.069269 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.069255 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 15 11:38:35.069483 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.069465 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 15 11:38:35.069674 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.069655 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 15 11:38:35.070046 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.070026 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fql47\"" Sep 15 11:38:35.071338 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.071324 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 15 11:38:35.072371 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.071901 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lvncp\"" Sep 15 11:38:35.072371 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.072191 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 15 11:38:35.072868 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.072848 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.075117 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.074922 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 15 11:38:35.075117 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.074941 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 15 11:38:35.075262 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.075222 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gjgj\"" Sep 15 11:38:35.075303 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.075283 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 15 11:38:35.075803 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.075558 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.077231 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.076498 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 15 11:38:35.077618 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.077594 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:38:35.077908 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.077889 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 15 11:38:35.078450 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.078412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.079361 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.079341 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lvzb7\"" Sep 15 11:38:35.079632 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.079616 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:38:35.080988 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.080969 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rsrml\"" Sep 15 11:38:35.081194 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.081179 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 15 11:38:35.081408 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.081387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:35.083286 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.083268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.083383 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.083323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 15 11:38:35.083383 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.083329 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 15 11:38:35.083486 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.083342 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:35.083631 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.083614 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dfnd5\"" Sep 15 11:38:35.086287 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.086269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:35.086381 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.086330 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:35.088245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-modprobe-d\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.088328 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysctl-d\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.088328 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-var-lib-kubelet\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.088328 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-log-socket\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.088426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovnkube-script-lib\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.088426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59x2\" (UniqueName: \"kubernetes.io/projected/99912355-5a7e-4bae-9df2-d1a431b0b5fe-kube-api-access-j59x2\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.088426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-kubernetes\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.088426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-tuned\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.088605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.088605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-registration-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.088605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-k8s-cni-cncf-io\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.088605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088533 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysctl-conf\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.088605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-etc-kubernetes\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.088605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99912355-5a7e-4bae-9df2-d1a431b0b5fe-hosts-file\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.088605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-tmp\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.088858 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-slash\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.088858 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-cni-bin\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.088858 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw8nk\" (UniqueName: \"kubernetes.io/projected/ac35be96-57a4-431c-9d2c-53b434b6efb4-kube-api-access-xw8nk\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.088858 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.088787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:35.088858 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.088844 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:35.089373 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089354 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 15 11:38:35.089454 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.089506 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-cni-bin\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.089506 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-conf-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.089599 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-systemd\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.089599 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-kubelet\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.089599 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-sys-fs\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.089599 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-hostroot\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.089788 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/862c12a2-77dc-498d-a877-21a39132727c-multus-daemon-config\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.089788 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-node-log\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.089788 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-multus-certs\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.089788 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.089788 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86bfea02-4d63-45f1-a3ca-d734e351199c-iptables-alerter-script\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.089788 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-var-lib-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-ovn\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/840cfbe9-71a6-4d47-b285-292f5462bb62-serviceca\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-socket-dir-parent\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-kubelet\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cni-binary-copy\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6bnr\" (UniqueName: \"kubernetes.io/projected/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-kube-api-access-c6bnr\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-systemd-units\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-cni-netd\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-socket-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.089990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-device-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86bfea02-4d63-45f1-a3ca-d734e351199c-host-slash\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-run-netns\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmfs2\" (UniqueName: \"kubernetes.io/projected/840cfbe9-71a6-4d47-b285-292f5462bb62-kube-api-access-bmfs2\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-sys\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-cni-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-cni-multus\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99912355-5a7e-4bae-9df2-d1a431b0b5fe-tmp-dir\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mgm\" (UniqueName: \"kubernetes.io/projected/86bfea02-4d63-45f1-a3ca-d734e351199c-kube-api-access-x2mgm\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cnibin\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-systemd\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovn-node-metrics-cert\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsf6b\" (UniqueName: \"kubernetes.io/projected/68466b36-8628-4c97-a7c4-871f3682d6bf-kube-api-access-fsf6b\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840cfbe9-71a6-4d47-b285-292f5462bb62-host\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-lib-modules\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fkk5\" (UniqueName: \"kubernetes.io/projected/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-kube-api-access-4fkk5\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.090510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-system-cni-dir\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090490 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-etc-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-env-overrides\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-system-cni-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysconfig\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090652 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-run\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-host\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090702 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovnkube-config\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-cnibin\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/862c12a2-77dc-498d-a877-21a39132727c-cni-binary-copy\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-netns\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.090951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.090799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bw8c\" (UniqueName: \"kubernetes.io/projected/862c12a2-77dc-498d-a877-21a39132727c-kube-api-access-7bw8c\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.116658 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.116631 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h9vh4" Sep 15 11:38:35.125618 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.125544 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h9vh4" Sep 15 11:38:35.178734 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.178706 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 15 11:38:35.190733 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.188269 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" event={"ID":"8319835104d09724226f87792cc29979","Type":"ContainerStarted","Data":"cb81a96aebd57eb8acc96d0a25e2ea817f049cf8bf231346f6b8efc28b1fcc6e"} Sep 15 11:38:35.190733 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.189977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" event={"ID":"2d02dff20630896118b374c2cd0a78b5","Type":"ContainerStarted","Data":"23245325250b393fa513053aa9ae2fe39f9c779d29dec82171ec3eebf74489b2"} Sep 15 11:38:35.191315 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-node-log\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.191440 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-node-log\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.191440 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-multus-certs\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.191440 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.191440 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86bfea02-4d63-45f1-a3ca-d734e351199c-iptables-alerter-script\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-var-lib-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191477 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-multus-certs\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-ovn\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191518 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-ovn\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/840cfbe9-71a6-4d47-b285-292f5462bb62-serviceca\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-socket-dir-parent\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-kubelet\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cni-binary-copy\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6bnr\" (UniqueName: \"kubernetes.io/projected/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-kube-api-access-c6bnr\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.191648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-systemd-units\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-cni-netd\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-socket-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-device-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1acb7e52-2368-4c5c-b3e5-c65c32136759-konnectivity-ca\") pod \"konnectivity-agent-k5k2z\" (UID: \"1acb7e52-2368-4c5c-b3e5-c65c32136759\") " pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-dbus\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86bfea02-4d63-45f1-a3ca-d734e351199c-host-slash\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-run-netns\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmfs2\" (UniqueName: \"kubernetes.io/projected/840cfbe9-71a6-4d47-b285-292f5462bb62-kube-api-access-bmfs2\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-sys\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/840cfbe9-71a6-4d47-b285-292f5462bb62-serviceca\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.191968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-cni-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-cni-multus\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99912355-5a7e-4bae-9df2-d1a431b0b5fe-tmp-dir\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.192172 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mgm\" (UniqueName: \"kubernetes.io/projected/86bfea02-4d63-45f1-a3ca-d734e351199c-kube-api-access-x2mgm\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cnibin\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-systemd\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192244 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-systemd-units\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-var-lib-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-device-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86bfea02-4d63-45f1-a3ca-d734e351199c-host-slash\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192385 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-socket-dir-parent\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192426 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-run-netns\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-kubelet\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-cni-netd\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192037 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-socket-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/86bfea02-4d63-45f1-a3ca-d734e351199c-iptables-alerter-script\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192496 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cni-binary-copy\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.193033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192521 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99912355-5a7e-4bae-9df2-d1a431b0b5fe-tmp-dir\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192525 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192535 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-cni-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-sys\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-cni-multus\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cnibin\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-run-systemd\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovn-node-metrics-cert\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsf6b\" (UniqueName: \"kubernetes.io/projected/68466b36-8628-4c97-a7c4-871f3682d6bf-kube-api-access-fsf6b\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192669 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840cfbe9-71a6-4d47-b285-292f5462bb62-host\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvm5t\" (UniqueName: \"kubernetes.io/projected/0e68ffcd-641e-4bbf-9703-5397e737b4b6-kube-api-access-zvm5t\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192760 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-lib-modules\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fkk5\" (UniqueName: \"kubernetes.io/projected/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-kube-api-access-4fkk5\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-system-cni-dir\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-etc-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.193881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-env-overrides\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192921 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-lib-modules\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-etc-openvswitch\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-system-cni-dir\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-system-cni-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysconfig\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-run\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-host\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193063 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysconfig\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovnkube-config\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-cnibin\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-system-cni-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193254 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-run\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193261 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193319 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-host\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.194806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193385 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-cnibin\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/862c12a2-77dc-498d-a877-21a39132727c-cni-binary-copy\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-netns\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193499 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bw8c\" (UniqueName: \"kubernetes.io/projected/862c12a2-77dc-498d-a877-21a39132727c-kube-api-access-7bw8c\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193531 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-env-overrides\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-netns\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-modprobe-d\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysctl-d\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-var-lib-kubelet\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovnkube-config\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193618 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-log-socket\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-log-socket\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovnkube-script-lib\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59x2\" (UniqueName: \"kubernetes.io/projected/99912355-5a7e-4bae-9df2-d1a431b0b5fe-kube-api-access-j59x2\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-var-lib-kubelet\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-kubelet-config\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.195678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-kubernetes\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-tuned\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193776 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysctl-d\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-kubernetes\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-modprobe-d\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.193977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-registration-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-k8s-cni-cncf-io\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysctl-conf\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-etc-kubernetes\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99912355-5a7e-4bae-9df2-d1a431b0b5fe-hosts-file\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovnkube-script-lib\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1acb7e52-2368-4c5c-b3e5-c65c32136759-agent-certs\") pod \"konnectivity-agent-k5k2z\" (UID: \"1acb7e52-2368-4c5c-b3e5-c65c32136759\") " pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-run-k8s-cni-cncf-io\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-tmp\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194224 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/862c12a2-77dc-498d-a877-21a39132727c-cni-binary-copy\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-slash\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.196662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-cni-bin\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194274 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-registration-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194274 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-sysctl-conf\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw8nk\" (UniqueName: \"kubernetes.io/projected/ac35be96-57a4-431c-9d2c-53b434b6efb4-kube-api-access-xw8nk\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194306 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-etc-kubernetes\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/99912355-5a7e-4bae-9df2-d1a431b0b5fe-hosts-file\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-cni-bin\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-conf-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-cni-bin\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194432 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194440 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-systemd\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-systemd\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194464 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-host-var-lib-cni-bin\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.192770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/840cfbe9-71a6-4d47-b285-292f5462bb62-host\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-kubelet\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-slash\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-sys-fs\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.197563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194513 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-multus-conf-dir\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-hostroot\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68466b36-8628-4c97-a7c4-871f3682d6bf-sys-fs\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-kubelet\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/862c12a2-77dc-498d-a877-21a39132727c-multus-daemon-config\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194577 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/862c12a2-77dc-498d-a877-21a39132727c-hostroot\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.194674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac35be96-57a4-431c-9d2c-53b434b6efb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.195012 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/862c12a2-77dc-498d-a877-21a39132727c-multus-daemon-config\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.197712 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-etc-tuned\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.197839 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ac35be96-57a4-431c-9d2c-53b434b6efb4-ovn-node-metrics-cert\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.198377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.198115 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-tmp\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.203803 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.201932 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mgm\" (UniqueName: \"kubernetes.io/projected/86bfea02-4d63-45f1-a3ca-d734e351199c-kube-api-access-x2mgm\") pod \"iptables-alerter-nvknt\" (UID: \"86bfea02-4d63-45f1-a3ca-d734e351199c\") " pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.203803 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.203742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw8nk\" (UniqueName: \"kubernetes.io/projected/ac35be96-57a4-431c-9d2c-53b434b6efb4-kube-api-access-xw8nk\") pod \"ovnkube-node-jhwc4\" (UID: \"ac35be96-57a4-431c-9d2c-53b434b6efb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.203803 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.203773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6bnr\" (UniqueName: \"kubernetes.io/projected/e13e3cca-b3e1-4760-be27-3bdbaf1f2512-kube-api-access-c6bnr\") pod \"multus-additional-cni-plugins-7j5wd\" (UID: \"e13e3cca-b3e1-4760-be27-3bdbaf1f2512\") " pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.204661 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.204636 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59x2\" (UniqueName: \"kubernetes.io/projected/99912355-5a7e-4bae-9df2-d1a431b0b5fe-kube-api-access-j59x2\") pod \"node-resolver-2kk9v\" (UID: \"99912355-5a7e-4bae-9df2-d1a431b0b5fe\") " pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.205783 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.205762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bw8c\" (UniqueName: \"kubernetes.io/projected/862c12a2-77dc-498d-a877-21a39132727c-kube-api-access-7bw8c\") pod \"multus-qrf5k\" (UID: \"862c12a2-77dc-498d-a877-21a39132727c\") " pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.205996 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.205977 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsf6b\" (UniqueName: \"kubernetes.io/projected/68466b36-8628-4c97-a7c4-871f3682d6bf-kube-api-access-fsf6b\") pod \"aws-ebs-csi-driver-node-n5nw4\" (UID: \"68466b36-8628-4c97-a7c4-871f3682d6bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.218868 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.218843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fkk5\" (UniqueName: \"kubernetes.io/projected/0d9926cb-38e7-4a29-8389-8cbad2ffb9b5-kube-api-access-4fkk5\") pod \"tuned-5bzbl\" (UID: \"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5\") " pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.223556 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.223535 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmfs2\" (UniqueName: \"kubernetes.io/projected/840cfbe9-71a6-4d47-b285-292f5462bb62-kube-api-access-bmfs2\") pod \"node-ca-69tt7\" (UID: \"840cfbe9-71a6-4d47-b285-292f5462bb62\") " pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.294841 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.294800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1acb7e52-2368-4c5c-b3e5-c65c32136759-konnectivity-ca\") pod \"konnectivity-agent-k5k2z\" (UID: \"1acb7e52-2368-4c5c-b3e5-c65c32136759\") " pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:35.295016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.294848 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-dbus\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.295016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.294877 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:35.295016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.294905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:35.295016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.294933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvm5t\" (UniqueName: \"kubernetes.io/projected/0e68ffcd-641e-4bbf-9703-5397e737b4b6-kube-api-access-zvm5t\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:35.295016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.294958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.295016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.294996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-kubelet-config\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.295016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.294996 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-dbus\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.295374 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.295025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1acb7e52-2368-4c5c-b3e5-c65c32136759-agent-certs\") pod \"konnectivity-agent-k5k2z\" (UID: \"1acb7e52-2368-4c5c-b3e5-c65c32136759\") " pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:35.295374 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.295035 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:35.295374 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.295077 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:35.295374 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.295117 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs podName:0e68ffcd-641e-4bbf-9703-5397e737b4b6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:35.795096142 +0000 UTC m=+3.140204751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs") pod "network-metrics-daemon-q7vl8" (UID: "0e68ffcd-641e-4bbf-9703-5397e737b4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:35.295374 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.295143 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret podName:72de5f15-c37b-4c5b-ab3b-e182a42b06ab nodeName:}" failed. No retries permitted until 2026-09-15 11:38:35.795127447 +0000 UTC m=+3.140236031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret") pod "global-pull-secret-syncer-p469s" (UID: "72de5f15-c37b-4c5b-ab3b-e182a42b06ab") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:35.295374 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.295220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-kubelet-config\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.295627 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.295421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1acb7e52-2368-4c5c-b3e5-c65c32136759-konnectivity-ca\") pod \"konnectivity-agent-k5k2z\" (UID: \"1acb7e52-2368-4c5c-b3e5-c65c32136759\") " pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:35.297935 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.297914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1acb7e52-2368-4c5c-b3e5-c65c32136759-agent-certs\") pod \"konnectivity-agent-k5k2z\" (UID: \"1acb7e52-2368-4c5c-b3e5-c65c32136759\") " pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:35.309324 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.309268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvm5t\" (UniqueName: \"kubernetes.io/projected/0e68ffcd-641e-4bbf-9703-5397e737b4b6-kube-api-access-zvm5t\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:35.309324 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.309317 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:38:35.309471 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.309336 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:38:35.309471 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.309347 2579 projected.go:196] Error preparing data for projected volume kube-api-access-62kx7 for pod openshift-network-diagnostics/network-check-target-4ztt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:35.309471 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.309405 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7 podName:6ac50613-e1b5-4278-9430-e64618567054 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:35.809392947 +0000 UTC m=+3.154501552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-62kx7" (UniqueName: "kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7") pod "network-check-target-4ztt5" (UID: "6ac50613-e1b5-4278-9430-e64618567054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:35.372931 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.372885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" Sep 15 11:38:35.383215 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.383169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nvknt" Sep 15 11:38:35.399180 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.399153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:38:35.404251 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.404230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" Sep 15 11:38:35.412069 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.412039 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2kk9v" Sep 15 11:38:35.419812 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.419779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-69tt7" Sep 15 11:38:35.427512 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.427492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" Sep 15 11:38:35.433215 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.433180 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qrf5k" Sep 15 11:38:35.438824 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.438804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:35.799113 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.799032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:35.799113 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.799066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:35.799327 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.799180 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:35.799327 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.799181 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:35.799327 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.799244 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret podName:72de5f15-c37b-4c5b-ab3b-e182a42b06ab nodeName:}" failed. No retries permitted until 2026-09-15 11:38:36.799231724 +0000 UTC m=+4.144340307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret") pod "global-pull-secret-syncer-p469s" (UID: "72de5f15-c37b-4c5b-ab3b-e182a42b06ab") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:35.799327 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.799259 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs podName:0e68ffcd-641e-4bbf-9703-5397e737b4b6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:36.799253145 +0000 UTC m=+4.144361730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs") pod "network-metrics-daemon-q7vl8" (UID: "0e68ffcd-641e-4bbf-9703-5397e737b4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:35.899795 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:35.899755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:35.899962 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.899932 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:38:35.899962 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.899950 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:38:35.899962 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.899959 2579 projected.go:196] Error preparing data for projected volume kube-api-access-62kx7 for pod openshift-network-diagnostics/network-check-target-4ztt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:35.900131 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:35.900038 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7 podName:6ac50613-e1b5-4278-9430-e64618567054 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:36.900022288 +0000 UTC m=+4.245130875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-62kx7" (UniqueName: "kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7") pod "network-check-target-4ztt5" (UID: "6ac50613-e1b5-4278-9430-e64618567054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:36.129856 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.129825 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:33:35 +0000 UTC" deadline="2028-05-12 03:36:21.141987821 +0000 UTC" Sep 15 11:38:36.129856 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.129856 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14511h57m45.012134781s" Sep 15 11:38:36.138319 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.138285 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9926cb_38e7_4a29_8389_8cbad2ffb9b5.slice/crio-4f310c4faae6e9268cabb450e08889f8dda122289c97ef122e792a56286a44c6 WatchSource:0}: Error finding container 4f310c4faae6e9268cabb450e08889f8dda122289c97ef122e792a56286a44c6: Status 404 returned error can't find the container with id 4f310c4faae6e9268cabb450e08889f8dda122289c97ef122e792a56286a44c6 Sep 15 11:38:36.140935 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.140872 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68466b36_8628_4c97_a7c4_871f3682d6bf.slice/crio-d6f4caeb003ef01cdc5388ea01dfd6f3f9910974404e25d569ef586c3221b634 WatchSource:0}: Error finding container d6f4caeb003ef01cdc5388ea01dfd6f3f9910974404e25d569ef586c3221b634: Status 404 returned error can't find the container with id d6f4caeb003ef01cdc5388ea01dfd6f3f9910974404e25d569ef586c3221b634 Sep 15 11:38:36.145274 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.145253 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bfea02_4d63_45f1_a3ca_d734e351199c.slice/crio-5f57fb803ab6d24ad7250ce94f4146c783341ebdfec42db339a06b0078126083 WatchSource:0}: Error finding container 5f57fb803ab6d24ad7250ce94f4146c783341ebdfec42db339a06b0078126083: Status 404 returned error can't find the container with id 5f57fb803ab6d24ad7250ce94f4146c783341ebdfec42db339a06b0078126083 Sep 15 11:38:36.146636 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.146614 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862c12a2_77dc_498d_a877_21a39132727c.slice/crio-451d7ef1c79cc182eaeadee8ac4391ed0676bf3150e84d43df34577f8a4aebee WatchSource:0}: Error finding container 451d7ef1c79cc182eaeadee8ac4391ed0676bf3150e84d43df34577f8a4aebee: Status 404 returned error can't find the container with id 451d7ef1c79cc182eaeadee8ac4391ed0676bf3150e84d43df34577f8a4aebee Sep 15 11:38:36.147387 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.147369 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13e3cca_b3e1_4760_be27_3bdbaf1f2512.slice/crio-767a553b9c6eb2dc917ee08744f4a1e220d8b03ad8a325a68c55a6888fce087e WatchSource:0}: Error finding container 767a553b9c6eb2dc917ee08744f4a1e220d8b03ad8a325a68c55a6888fce087e: Status 404 returned error can't find the container with id 767a553b9c6eb2dc917ee08744f4a1e220d8b03ad8a325a68c55a6888fce087e Sep 15 11:38:36.148406 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.148219 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac35be96_57a4_431c_9d2c_53b434b6efb4.slice/crio-2b0e059df1ddaa14f21b0e960014591fb08819836095f0e331ff13ef30f14c2e WatchSource:0}: Error finding container 2b0e059df1ddaa14f21b0e960014591fb08819836095f0e331ff13ef30f14c2e: Status 404 returned error can't find the container with id 2b0e059df1ddaa14f21b0e960014591fb08819836095f0e331ff13ef30f14c2e Sep 15 11:38:36.150185 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.149291 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840cfbe9_71a6_4d47_b285_292f5462bb62.slice/crio-a9b114bf666215dfb9657b558590283f6c50c0b614541fefe0981adf69202c78 WatchSource:0}: Error finding container a9b114bf666215dfb9657b558590283f6c50c0b614541fefe0981adf69202c78: Status 404 returned error can't find the container with id a9b114bf666215dfb9657b558590283f6c50c0b614541fefe0981adf69202c78 Sep 15 11:38:36.150891 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.150870 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1acb7e52_2368_4c5c_b3e5_c65c32136759.slice/crio-d873260d688aeb18e18b75173e80f344ca5645dac11900af4d50f310f8ee0a1d WatchSource:0}: Error finding container d873260d688aeb18e18b75173e80f344ca5645dac11900af4d50f310f8ee0a1d: Status 404 returned error can't find the container with id d873260d688aeb18e18b75173e80f344ca5645dac11900af4d50f310f8ee0a1d Sep 15 11:38:36.173863 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:38:36.173834 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99912355_5a7e_4bae_9df2_d1a431b0b5fe.slice/crio-04b13386e81f0c2b41081e243302a4453db69f5a2794ba2ff507046095918166 WatchSource:0}: Error finding container 04b13386e81f0c2b41081e243302a4453db69f5a2794ba2ff507046095918166: Status 404 returned error can't find the container with id 04b13386e81f0c2b41081e243302a4453db69f5a2794ba2ff507046095918166 Sep 15 11:38:36.195540 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.195491 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qrf5k" event={"ID":"862c12a2-77dc-498d-a877-21a39132727c","Type":"ContainerStarted","Data":"451d7ef1c79cc182eaeadee8ac4391ed0676bf3150e84d43df34577f8a4aebee"} Sep 15 11:38:36.196679 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.196657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k5k2z" event={"ID":"1acb7e52-2368-4c5c-b3e5-c65c32136759","Type":"ContainerStarted","Data":"d873260d688aeb18e18b75173e80f344ca5645dac11900af4d50f310f8ee0a1d"} Sep 15 11:38:36.197901 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.197872 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"2b0e059df1ddaa14f21b0e960014591fb08819836095f0e331ff13ef30f14c2e"} Sep 15 11:38:36.198996 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.198969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" event={"ID":"e13e3cca-b3e1-4760-be27-3bdbaf1f2512","Type":"ContainerStarted","Data":"767a553b9c6eb2dc917ee08744f4a1e220d8b03ad8a325a68c55a6888fce087e"} Sep 15 11:38:36.200016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.199991 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" event={"ID":"68466b36-8628-4c97-a7c4-871f3682d6bf","Type":"ContainerStarted","Data":"d6f4caeb003ef01cdc5388ea01dfd6f3f9910974404e25d569ef586c3221b634"} Sep 15 11:38:36.201353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.201321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2kk9v" event={"ID":"99912355-5a7e-4bae-9df2-d1a431b0b5fe","Type":"ContainerStarted","Data":"04b13386e81f0c2b41081e243302a4453db69f5a2794ba2ff507046095918166"} Sep 15 11:38:36.202856 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.202836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-69tt7" event={"ID":"840cfbe9-71a6-4d47-b285-292f5462bb62","Type":"ContainerStarted","Data":"a9b114bf666215dfb9657b558590283f6c50c0b614541fefe0981adf69202c78"} Sep 15 11:38:36.203943 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.203919 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nvknt" event={"ID":"86bfea02-4d63-45f1-a3ca-d734e351199c","Type":"ContainerStarted","Data":"5f57fb803ab6d24ad7250ce94f4146c783341ebdfec42db339a06b0078126083"} Sep 15 11:38:36.204975 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.204957 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" event={"ID":"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5","Type":"ContainerStarted","Data":"4f310c4faae6e9268cabb450e08889f8dda122289c97ef122e792a56286a44c6"} Sep 15 11:38:36.224666 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.224485 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 15 11:38:36.805557 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.805452 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:36.805557 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.805516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:36.805778 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:36.805666 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:36.805778 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:36.805730 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret podName:72de5f15-c37b-4c5b-ab3b-e182a42b06ab nodeName:}" failed. No retries permitted until 2026-09-15 11:38:38.805712327 +0000 UTC m=+6.150820934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret") pod "global-pull-secret-syncer-p469s" (UID: "72de5f15-c37b-4c5b-ab3b-e182a42b06ab") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:36.806144 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:36.806125 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:36.806228 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:36.806182 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs podName:0e68ffcd-641e-4bbf-9703-5397e737b4b6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:38.806166068 +0000 UTC m=+6.151274672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs") pod "network-metrics-daemon-q7vl8" (UID: "0e68ffcd-641e-4bbf-9703-5397e737b4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:36.907385 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:36.906733 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:36.907385 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:36.906906 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:38:36.907385 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:36.906930 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:38:36.907385 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:36.906943 2579 projected.go:196] Error preparing data for projected volume kube-api-access-62kx7 for pod openshift-network-diagnostics/network-check-target-4ztt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:36.907385 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:36.907000 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7 podName:6ac50613-e1b5-4278-9430-e64618567054 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:38.906983111 +0000 UTC m=+6.252091703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-62kx7" (UniqueName: "kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7") pod "network-check-target-4ztt5" (UID: "6ac50613-e1b5-4278-9430-e64618567054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:37.130654 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:37.130545 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:33:35 +0000 UTC" deadline="2028-04-17 13:49:31.266846116 +0000 UTC" Sep 15 11:38:37.130654 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:37.130604 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13922h10m54.13624603s" Sep 15 11:38:37.182560 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:37.182526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:37.182741 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:37.182672 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:37.183096 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:37.183071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:37.183192 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:37.183165 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:37.183291 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:37.183266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:37.183374 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:37.183355 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:37.224541 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:37.224503 2579 generic.go:356] "Generic (PLEG): container finished" podID="8319835104d09724226f87792cc29979" containerID="dfc93c0728c2d53eb7cb362fc2ab967ab55edf4c942acb90cb17f100d1a73ba7" exitCode=0 Sep 15 11:38:37.224724 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:37.224617 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" event={"ID":"8319835104d09724226f87792cc29979","Type":"ContainerDied","Data":"dfc93c0728c2d53eb7cb362fc2ab967ab55edf4c942acb90cb17f100d1a73ba7"} Sep 15 11:38:37.234183 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:37.234126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" event={"ID":"2d02dff20630896118b374c2cd0a78b5","Type":"ContainerStarted","Data":"8a04a2a1178f249cec8cd38c75cb01ecd9f7e6139111a1f7ab7c3668926193cf"} Sep 15 11:38:38.268039 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:38.267765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" event={"ID":"8319835104d09724226f87792cc29979","Type":"ContainerStarted","Data":"172cb4032a7d8ff6e7ebd1a8d9b150b88af374d4aab1f09391a4e57e4ff43047"} Sep 15 11:38:38.283668 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:38.283611 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-16.ec2.internal" podStartSLOduration=4.283588826 podStartE2EDuration="4.283588826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:37.259458942 +0000 UTC m=+4.604567549" watchObservedRunningTime="2026-09-15 11:38:38.283588826 +0000 UTC m=+5.628697433" Sep 15 11:38:38.826538 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:38.825888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:38.826538 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:38.825941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:38.826538 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:38.826079 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:38.826538 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:38.826081 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:38.826538 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:38.826151 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret podName:72de5f15-c37b-4c5b-ab3b-e182a42b06ab nodeName:}" failed. No retries permitted until 2026-09-15 11:38:42.826132759 +0000 UTC m=+10.171241343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret") pod "global-pull-secret-syncer-p469s" (UID: "72de5f15-c37b-4c5b-ab3b-e182a42b06ab") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:38.826538 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:38.826171 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs podName:0e68ffcd-641e-4bbf-9703-5397e737b4b6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:42.826159834 +0000 UTC m=+10.171268423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs") pod "network-metrics-daemon-q7vl8" (UID: "0e68ffcd-641e-4bbf-9703-5397e737b4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:38.927124 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:38.927058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:38.932541 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:38.932506 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:38:38.932686 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:38.932550 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:38:38.932686 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:38.932574 2579 projected.go:196] Error preparing data for projected volume kube-api-access-62kx7 for pod openshift-network-diagnostics/network-check-target-4ztt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:38.932686 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:38.932648 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7 podName:6ac50613-e1b5-4278-9430-e64618567054 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:42.932620299 +0000 UTC m=+10.277728886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-62kx7" (UniqueName: "kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7") pod "network-check-target-4ztt5" (UID: "6ac50613-e1b5-4278-9430-e64618567054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:39.182850 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:39.182252 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:39.182850 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:39.182382 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:39.182850 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:39.182502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:39.182850 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:39.182502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:39.182850 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:39.182622 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:39.182850 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:39.182682 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:41.182090 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:41.182047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:41.182568 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:41.182177 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:41.182568 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:41.182539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:41.182713 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:41.182646 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:41.182713 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:41.182694 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:41.182782 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:41.182737 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:42.857432 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:42.857396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:42.857846 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:42.857443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:42.857846 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:42.857683 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:42.857846 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:42.857758 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret podName:72de5f15-c37b-4c5b-ab3b-e182a42b06ab nodeName:}" failed. No retries permitted until 2026-09-15 11:38:50.857732458 +0000 UTC m=+18.202841046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret") pod "global-pull-secret-syncer-p469s" (UID: "72de5f15-c37b-4c5b-ab3b-e182a42b06ab") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:42.858193 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:42.858174 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:42.858347 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:42.858314 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs podName:0e68ffcd-641e-4bbf-9703-5397e737b4b6 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:50.85827891 +0000 UTC m=+18.203387497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs") pod "network-metrics-daemon-q7vl8" (UID: "0e68ffcd-641e-4bbf-9703-5397e737b4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:42.958252 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:42.958187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:42.958488 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:42.958363 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:38:42.958488 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:42.958387 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:38:42.958488 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:42.958399 2579 projected.go:196] Error preparing data for projected volume kube-api-access-62kx7 for pod openshift-network-diagnostics/network-check-target-4ztt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:42.958488 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:42.958459 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7 podName:6ac50613-e1b5-4278-9430-e64618567054 nodeName:}" failed. No retries permitted until 2026-09-15 11:38:50.958441899 +0000 UTC m=+18.303550506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-62kx7" (UniqueName: "kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7") pod "network-check-target-4ztt5" (UID: "6ac50613-e1b5-4278-9430-e64618567054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:43.183273 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:43.183176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:43.183444 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:43.183303 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:43.183444 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:43.183316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:43.183444 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:43.183431 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:43.183605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:43.183478 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:43.183605 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:43.183564 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:45.182417 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:45.182380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:45.182894 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:45.182380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:45.182894 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:45.182513 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:45.182894 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:45.182528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:45.182894 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:45.182638 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:45.182894 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:45.182710 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:47.182216 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:47.181721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:47.182216 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:47.181755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:47.182216 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:47.181836 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:47.182216 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:47.181736 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:47.182216 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:47.181974 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:47.182216 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:47.182066 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:49.182364 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:49.182327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:49.182364 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:49.182365 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:49.182855 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:49.182334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:49.182855 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:49.182474 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:49.182855 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:49.182563 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:49.182855 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:49.182661 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:50.920784 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:50.920750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:50.920784 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:50.920797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:50.921443 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:50.920917 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:50.921443 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:50.920929 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:50.921443 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:50.920976 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret podName:72de5f15-c37b-4c5b-ab3b-e182a42b06ab nodeName:}" failed. No retries permitted until 2026-09-15 11:39:06.920962445 +0000 UTC m=+34.266071029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret") pod "global-pull-secret-syncer-p469s" (UID: "72de5f15-c37b-4c5b-ab3b-e182a42b06ab") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:38:50.921443 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:50.921003 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs podName:0e68ffcd-641e-4bbf-9703-5397e737b4b6 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:06.920983345 +0000 UTC m=+34.266091946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs") pod "network-metrics-daemon-q7vl8" (UID: "0e68ffcd-641e-4bbf-9703-5397e737b4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:38:51.021725 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:51.021675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:51.021894 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:51.021867 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:38:51.021936 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:51.021896 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:38:51.021936 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:51.021910 2579 projected.go:196] Error preparing data for projected volume kube-api-access-62kx7 for pod openshift-network-diagnostics/network-check-target-4ztt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:51.022025 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:51.021978 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7 podName:6ac50613-e1b5-4278-9430-e64618567054 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:07.02195783 +0000 UTC m=+34.367066418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-62kx7" (UniqueName: "kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7") pod "network-check-target-4ztt5" (UID: "6ac50613-e1b5-4278-9430-e64618567054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:38:51.181689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:51.181598 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:51.181689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:51.181619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:51.181689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:51.181593 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:51.181952 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:51.181727 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:51.181952 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:51.181799 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:51.181952 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:51.181868 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:53.183002 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:53.182969 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:53.183402 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:53.183081 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:53.183402 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:53.183099 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:53.183402 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:53.183189 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:53.183402 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:53.183282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:53.183402 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:53.183380 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:55.182348 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.182121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:55.183034 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.182138 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:55.183034 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:55.182396 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:55.183034 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.182170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:55.183034 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:55.182449 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:55.183034 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:55.182536 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:55.304905 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.304451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2kk9v" event={"ID":"99912355-5a7e-4bae-9df2-d1a431b0b5fe","Type":"ContainerStarted","Data":"99fdd740fb046c0f1500792915a6e37dcddf3b9dfe289ed2b9ecd3bd209e63db"} Sep 15 11:38:55.306122 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.306087 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-69tt7" event={"ID":"840cfbe9-71a6-4d47-b285-292f5462bb62","Type":"ContainerStarted","Data":"22483098bad8e2bf824604b307e5611f0a5a00623797242cdb128381cc53fd5a"} Sep 15 11:38:55.308163 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.307854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" event={"ID":"0d9926cb-38e7-4a29-8389-8cbad2ffb9b5","Type":"ContainerStarted","Data":"65db7e3dc1cb0d29a87ab2e30a359111c6432bc6ac7fb88c774ef275cb142932"} Sep 15 11:38:55.309522 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.309498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qrf5k" event={"ID":"862c12a2-77dc-498d-a877-21a39132727c","Type":"ContainerStarted","Data":"bcdd00d2a8aadc88fb17d3da81db41cb318023b268d0145a489f08197fc99075"} Sep 15 11:38:55.311621 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.311301 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k5k2z" event={"ID":"1acb7e52-2368-4c5c-b3e5-c65c32136759","Type":"ContainerStarted","Data":"9c5dff6dfd7546c09fcbd0cf1f289345a48c2963390d62adb0d197eb8ce6828c"} Sep 15 11:38:55.314888 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.314862 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"788624e85049d4a9630109291836bd2ef027a7269667bdbf266f93680fac57ff"} Sep 15 11:38:55.314987 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.314942 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"87134b91bfc031eec2fbb7451e35264b3ad7f7097ed6f41a245afe422bf2663f"} Sep 15 11:38:55.315045 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.314994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"398947f24ce9e2e48779311299c93119c90c077fac26fe033ca6a7186c5ed331"} Sep 15 11:38:55.315104 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.315046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"87f3a30ba64a9bd9255cb8a519e89f44fd43e03336aa04f832514058ae4b01eb"} Sep 15 11:38:55.315104 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.315094 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"265908fc791b696f3f16dc00a0fbd53ec8df58a9280c730149b8304466da5467"} Sep 15 11:38:55.315196 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.315146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"69bfdfe24b026427c1b777fe4710fd756ae1b86e31015c18899a828c8fa1ec45"} Sep 15 11:38:55.317022 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.316996 2579 generic.go:356] "Generic (PLEG): container finished" podID="e13e3cca-b3e1-4760-be27-3bdbaf1f2512" containerID="e314a6c704bc53f28dc612197b92948ef6a70e1104380eec928035bf87e1920e" exitCode=0 Sep 15 11:38:55.317115 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.317069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" event={"ID":"e13e3cca-b3e1-4760-be27-3bdbaf1f2512","Type":"ContainerDied","Data":"e314a6c704bc53f28dc612197b92948ef6a70e1104380eec928035bf87e1920e"} Sep 15 11:38:55.320284 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.320244 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-16.ec2.internal" podStartSLOduration=21.320232422 podStartE2EDuration="21.320232422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:38.284810151 +0000 UTC m=+5.629918752" watchObservedRunningTime="2026-09-15 11:38:55.320232422 +0000 UTC m=+22.665341029" Sep 15 11:38:55.321250 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.321225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" event={"ID":"68466b36-8628-4c97-a7c4-871f3682d6bf","Type":"ContainerStarted","Data":"8079c8283925066cfa8a22696a3dd1cb189895733b0eff1cc43184486e025afa"} Sep 15 11:38:55.335864 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.335815 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2kk9v" podStartSLOduration=4.023675564 podStartE2EDuration="22.335798733s" totalImagesPullingTime="18.312123169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:55.320857391 +0000 UTC m=+22.665965996" watchObservedRunningTime="2026-09-15 11:38:55.335798733 +0000 UTC m=+22.680907340" Sep 15 11:38:55.335976 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.335888 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-69tt7" podStartSLOduration=4.107002007 podStartE2EDuration="22.335882014s" totalImagesPullingTime="18.228880007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:55.335463495 +0000 UTC m=+22.680572104" watchObservedRunningTime="2026-09-15 11:38:55.335882014 +0000 UTC m=+22.680990621" Sep 15 11:38:55.374891 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.374710 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qrf5k" podStartSLOduration=3.880836388 podStartE2EDuration="22.374689591s" totalImagesPullingTime="18.493853203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:55.353523048 +0000 UTC m=+22.698631654" watchObservedRunningTime="2026-09-15 11:38:55.374689591 +0000 UTC m=+22.719798206" Sep 15 11:38:55.393548 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:55.393474 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5bzbl" podStartSLOduration=4.045548933 podStartE2EDuration="22.393462362s" totalImagesPullingTime="18.347913429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:55.393288062 +0000 UTC m=+22.738396670" watchObservedRunningTime="2026-09-15 11:38:55.393462362 +0000 UTC m=+22.738570967" Sep 15 11:38:56.105302 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:56.105075 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 15 11:38:56.158044 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:56.157923 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T11:38:56.105291902Z","UUID":"fc537c16-0b19-4b50-93d5-0b0a96106e79","Handler":null,"Name":"","Endpoint":""} Sep 15 11:38:56.159593 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:56.159571 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 15 11:38:56.159705 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:56.159599 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 15 11:38:56.325886 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:56.325372 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nvknt" event={"ID":"86bfea02-4d63-45f1-a3ca-d734e351199c","Type":"ContainerStarted","Data":"36edfe645ff16fff5fcc3059fd193a01b1132e2da24b2249a19b439c71b3dd6f"} Sep 15 11:38:56.327530 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:56.327500 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" event={"ID":"68466b36-8628-4c97-a7c4-871f3682d6bf","Type":"ContainerStarted","Data":"b4dc8b4e784580458c9e8f2f2bdb595b95a89c32ae66cb383cf680eceeb1003f"} Sep 15 11:38:56.341181 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:56.341132 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nvknt" podStartSLOduration=5.001750871 podStartE2EDuration="23.341111686s" totalImagesPullingTime="18.339360815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:56.340840206 +0000 UTC m=+23.685948813" watchObservedRunningTime="2026-09-15 11:38:56.341111686 +0000 UTC m=+23.686220296" Sep 15 11:38:56.341496 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:56.341453 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k5k2z" podStartSLOduration=5.467029244 podStartE2EDuration="23.341437421s" totalImagesPullingTime="17.874408177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:55.411793396 +0000 UTC m=+22.756902003" watchObservedRunningTime="2026-09-15 11:38:56.341437421 +0000 UTC m=+23.686546029" Sep 15 11:38:57.182019 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:57.181980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:57.182157 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:57.181980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:57.182157 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:57.182103 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:57.182157 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:57.182101 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:57.182346 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:57.182175 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:38:57.182346 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:57.182288 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:57.334963 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:57.334917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"e20d018f79ca40ae56b2376b1dbd90c7900d59c133ee5efda7326ac86697faa8"} Sep 15 11:38:57.337859 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:57.337828 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" event={"ID":"68466b36-8628-4c97-a7c4-871f3682d6bf","Type":"ContainerStarted","Data":"44da8464115549a5af7e33ae5b155a5f3717843f074344867dad72fe990a45c4"} Sep 15 11:38:57.358927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:57.358603 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n5nw4" podStartSLOduration=3.733121121 podStartE2EDuration="24.358584504s" totalImagesPullingTime="20.625463383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:57.358054259 +0000 UTC m=+24.703162866" watchObservedRunningTime="2026-09-15 11:38:57.358584504 +0000 UTC m=+24.703693109" Sep 15 11:38:57.941083 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:57.941041 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:57.941848 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:57.941819 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:38:59.181954 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:59.181920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:38:59.182479 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:59.181920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:38:59.182479 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:59.182041 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:38:59.182479 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:38:59.181920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:38:59.182479 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:59.182138 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:38:59.182479 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:38:59.182227 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:39:00.349649 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:00.349285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" event={"ID":"ac35be96-57a4-431c-9d2c-53b434b6efb4","Type":"ContainerStarted","Data":"a04cf095a17bef8866e04e387f5a483a83f2c9f25c7761aa3169d08eabcc3a0e"} Sep 15 11:39:00.349649 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:00.349641 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:39:00.350316 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:00.349690 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:39:00.351494 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:00.351469 2579 generic.go:356] "Generic (PLEG): container finished" podID="e13e3cca-b3e1-4760-be27-3bdbaf1f2512" containerID="52a14abeb5f91446549f274e28364c93a239db3c85a3ed80c72d4fdfdcc5e915" exitCode=0 Sep 15 11:39:00.351569 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:00.351526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" event={"ID":"e13e3cca-b3e1-4760-be27-3bdbaf1f2512","Type":"ContainerDied","Data":"52a14abeb5f91446549f274e28364c93a239db3c85a3ed80c72d4fdfdcc5e915"} Sep 15 11:39:00.366233 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:00.366185 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:39:00.378908 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:00.378858 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" podStartSLOduration=9.028788461 podStartE2EDuration="27.378842595s" totalImagesPullingTime="18.350054134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:00.378338149 +0000 UTC m=+27.723446756" watchObservedRunningTime="2026-09-15 11:39:00.378842595 +0000 UTC m=+27.723951200" Sep 15 11:39:01.181701 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.181672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:01.181874 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.181668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:39:01.181874 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:01.181779 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:39:01.181874 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.181797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:01.182013 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:01.181896 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:39:01.182013 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:01.181960 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:39:01.353928 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.353895 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:39:01.368300 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.368276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:39:01.587507 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.587244 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:39:01.587663 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.587608 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 15 11:39:01.588098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.588070 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k5k2z" Sep 15 11:39:01.776816 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.776702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q7vl8"] Sep 15 11:39:01.776955 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.776826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:39:01.776955 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:01.776927 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:39:01.777534 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.777509 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4ztt5"] Sep 15 11:39:01.777646 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.777622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:01.777731 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:01.777713 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:39:01.788818 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.788789 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p469s"] Sep 15 11:39:01.788919 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:01.788885 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:01.789013 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:01.788982 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:39:02.360878 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:02.360846 2579 generic.go:356] "Generic (PLEG): container finished" podID="e13e3cca-b3e1-4760-be27-3bdbaf1f2512" containerID="a28526f9f250a25078d22bd821386c2b15290c9b48cf65831451e742cba6002c" exitCode=0 Sep 15 11:39:02.361465 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:02.360933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" event={"ID":"e13e3cca-b3e1-4760-be27-3bdbaf1f2512","Type":"ContainerDied","Data":"a28526f9f250a25078d22bd821386c2b15290c9b48cf65831451e742cba6002c"} Sep 15 11:39:03.182297 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:03.182267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:03.182297 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:03.182281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:39:03.182479 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:03.182387 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:39:03.182479 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:03.182469 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:39:03.365821 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:03.365789 2579 generic.go:356] "Generic (PLEG): container finished" podID="e13e3cca-b3e1-4760-be27-3bdbaf1f2512" containerID="cf0f459ea185c306f8162d26999ea61a7a365d604575dad73483816f77bf03db" exitCode=0 Sep 15 11:39:03.366212 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:03.365873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" event={"ID":"e13e3cca-b3e1-4760-be27-3bdbaf1f2512","Type":"ContainerDied","Data":"cf0f459ea185c306f8162d26999ea61a7a365d604575dad73483816f77bf03db"} Sep 15 11:39:04.182416 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:04.182333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:04.182574 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:04.182471 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:39:05.182408 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:05.182367 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:39:05.182876 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:05.182428 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:05.182876 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:05.182577 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:39:05.183020 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:05.182997 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:39:06.181972 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:06.181892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:06.182128 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:06.182015 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p469s" podUID="72de5f15-c37b-4c5b-ab3b-e182a42b06ab" Sep 15 11:39:06.944934 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:06.944900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:39:06.944934 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:06.944949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:06.945469 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:06.945070 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 15 11:39:06.945469 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:06.945069 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:39:06.945469 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:06.945138 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret podName:72de5f15-c37b-4c5b-ab3b-e182a42b06ab nodeName:}" failed. No retries permitted until 2026-09-15 11:39:38.945121069 +0000 UTC m=+66.290229655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret") pod "global-pull-secret-syncer-p469s" (UID: "72de5f15-c37b-4c5b-ab3b-e182a42b06ab") : object "kube-system"/"original-pull-secret" not registered Sep 15 11:39:06.945469 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:06.945158 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs podName:0e68ffcd-641e-4bbf-9703-5397e737b4b6 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:38.945147824 +0000 UTC m=+66.290256409 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs") pod "network-metrics-daemon-q7vl8" (UID: "0e68ffcd-641e-4bbf-9703-5397e737b4b6") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 15 11:39:07.045785 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.045745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:07.045958 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:07.045937 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 15 11:39:07.046001 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:07.045962 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 15 11:39:07.046001 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:07.045972 2579 projected.go:196] Error preparing data for projected volume kube-api-access-62kx7 for pod openshift-network-diagnostics/network-check-target-4ztt5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:39:07.046065 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:07.046025 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7 podName:6ac50613-e1b5-4278-9430-e64618567054 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.046011747 +0000 UTC m=+66.391120336 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-62kx7" (UniqueName: "kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7") pod "network-check-target-4ztt5" (UID: "6ac50613-e1b5-4278-9430-e64618567054") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 15 11:39:07.182351 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.182313 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:39:07.182537 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:07.182462 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q7vl8" podUID="0e68ffcd-641e-4bbf-9703-5397e737b4b6" Sep 15 11:39:07.182537 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.182524 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:07.182653 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:07.182637 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ztt5" podUID="6ac50613-e1b5-4278-9430-e64618567054" Sep 15 11:39:07.463939 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.463911 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-16.ec2.internal" event="NodeReady" Sep 15 11:39:07.464082 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.464050 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 15 11:39:07.508733 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.508690 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d29g9"] Sep 15 11:39:07.526487 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.526449 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qr6z5"] Sep 15 11:39:07.526671 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.526609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.529401 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.529379 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 15 11:39:07.529654 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.529639 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hx5hb\"" Sep 15 11:39:07.529815 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.529799 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 15 11:39:07.548540 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.548506 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qr6z5"] Sep 15 11:39:07.548675 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.548556 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d29g9"] Sep 15 11:39:07.548675 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.548638 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qr6z5" Sep 15 11:39:07.551763 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.551735 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 15 11:39:07.551763 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.551738 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vqwj6\"" Sep 15 11:39:07.552174 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.552155 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 15 11:39:07.552268 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.552189 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 15 11:39:07.649376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.649338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddba62f7-548c-494e-841d-e5ed366f84ce-config-volume\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.649376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.649374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddba62f7-548c-494e-841d-e5ed366f84ce-tmp-dir\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.649543 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.649394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c9ca50-20cf-40e7-a851-37f9e1488015-cert\") pod \"ingress-canary-qr6z5\" (UID: \"84c9ca50-20cf-40e7-a851-37f9e1488015\") " pod="openshift-ingress-canary/ingress-canary-qr6z5" Sep 15 11:39:07.649543 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.649434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5v95\" (UniqueName: \"kubernetes.io/projected/ddba62f7-548c-494e-841d-e5ed366f84ce-kube-api-access-l5v95\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.649543 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.649472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.649543 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.649512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ffv\" (UniqueName: \"kubernetes.io/projected/84c9ca50-20cf-40e7-a851-37f9e1488015-kube-api-access-t4ffv\") pod \"ingress-canary-qr6z5\" (UID: \"84c9ca50-20cf-40e7-a851-37f9e1488015\") " pod="openshift-ingress-canary/ingress-canary-qr6z5" Sep 15 11:39:07.750601 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.750573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ffv\" (UniqueName: \"kubernetes.io/projected/84c9ca50-20cf-40e7-a851-37f9e1488015-kube-api-access-t4ffv\") pod \"ingress-canary-qr6z5\" (UID: \"84c9ca50-20cf-40e7-a851-37f9e1488015\") " pod="openshift-ingress-canary/ingress-canary-qr6z5" Sep 15 11:39:07.750785 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.750632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddba62f7-548c-494e-841d-e5ed366f84ce-config-volume\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.750785 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.750650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddba62f7-548c-494e-841d-e5ed366f84ce-tmp-dir\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.750785 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.750666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c9ca50-20cf-40e7-a851-37f9e1488015-cert\") pod \"ingress-canary-qr6z5\" (UID: \"84c9ca50-20cf-40e7-a851-37f9e1488015\") " pod="openshift-ingress-canary/ingress-canary-qr6z5" Sep 15 11:39:07.750785 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.750681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5v95\" (UniqueName: \"kubernetes.io/projected/ddba62f7-548c-494e-841d-e5ed366f84ce-kube-api-access-l5v95\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.750785 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.750709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.750994 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:07.750802 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:39:07.750994 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:07.750881 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls podName:ddba62f7-548c-494e-841d-e5ed366f84ce nodeName:}" failed. No retries permitted until 2026-09-15 11:39:08.250862163 +0000 UTC m=+35.595970748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls") pod "dns-default-d29g9" (UID: "ddba62f7-548c-494e-841d-e5ed366f84ce") : secret "dns-default-metrics-tls" not found Sep 15 11:39:07.751279 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.751259 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddba62f7-548c-494e-841d-e5ed366f84ce-tmp-dir\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.751467 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.751449 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ddba62f7-548c-494e-841d-e5ed366f84ce-config-volume\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.755098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.755068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84c9ca50-20cf-40e7-a851-37f9e1488015-cert\") pod \"ingress-canary-qr6z5\" (UID: \"84c9ca50-20cf-40e7-a851-37f9e1488015\") " pod="openshift-ingress-canary/ingress-canary-qr6z5" Sep 15 11:39:07.760810 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.760778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5v95\" (UniqueName: \"kubernetes.io/projected/ddba62f7-548c-494e-841d-e5ed366f84ce-kube-api-access-l5v95\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:07.760903 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.760852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ffv\" (UniqueName: \"kubernetes.io/projected/84c9ca50-20cf-40e7-a851-37f9e1488015-kube-api-access-t4ffv\") pod \"ingress-canary-qr6z5\" (UID: \"84c9ca50-20cf-40e7-a851-37f9e1488015\") " pod="openshift-ingress-canary/ingress-canary-qr6z5" Sep 15 11:39:07.903245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:07.903140 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qr6z5" Sep 15 11:39:08.182057 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:08.181972 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:08.184653 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:08.184635 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 11:39:08.225692 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:08.225662 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qr6z5"] Sep 15 11:39:08.228874 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:39:08.228849 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c9ca50_20cf_40e7_a851_37f9e1488015.slice/crio-171fc39698fde015f6afac31490cdc5c89badbfc3798ea4e3c1685b9670bdfc4 WatchSource:0}: Error finding container 171fc39698fde015f6afac31490cdc5c89badbfc3798ea4e3c1685b9670bdfc4: Status 404 returned error can't find the container with id 171fc39698fde015f6afac31490cdc5c89badbfc3798ea4e3c1685b9670bdfc4 Sep 15 11:39:08.254189 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:08.254162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:08.254337 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:08.254317 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:39:08.254393 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:08.254384 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls podName:ddba62f7-548c-494e-841d-e5ed366f84ce nodeName:}" failed. No retries permitted until 2026-09-15 11:39:09.254369499 +0000 UTC m=+36.599478082 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls") pod "dns-default-d29g9" (UID: "ddba62f7-548c-494e-841d-e5ed366f84ce") : secret "dns-default-metrics-tls" not found Sep 15 11:39:08.383741 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:08.383711 2579 generic.go:356] "Generic (PLEG): container finished" podID="e13e3cca-b3e1-4760-be27-3bdbaf1f2512" containerID="805e7745c6eb21907e8c4df4a6a2abb99eec578da6b1e0f9c4e91e700014e06a" exitCode=0 Sep 15 11:39:08.383890 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:08.383746 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" event={"ID":"e13e3cca-b3e1-4760-be27-3bdbaf1f2512","Type":"ContainerDied","Data":"805e7745c6eb21907e8c4df4a6a2abb99eec578da6b1e0f9c4e91e700014e06a"} Sep 15 11:39:08.384984 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:08.384959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qr6z5" event={"ID":"84c9ca50-20cf-40e7-a851-37f9e1488015","Type":"ContainerStarted","Data":"171fc39698fde015f6afac31490cdc5c89badbfc3798ea4e3c1685b9670bdfc4"} Sep 15 11:39:09.182332 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.182297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:09.182886 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.182297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:39:09.185937 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.185901 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmnh2\"" Sep 15 11:39:09.186234 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.186193 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 11:39:09.186494 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.186478 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 11:39:09.186714 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.186698 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwddj\"" Sep 15 11:39:09.186934 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.186920 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 11:39:09.262565 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.262508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:09.262751 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:09.262629 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:39:09.262751 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:09.262719 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls podName:ddba62f7-548c-494e-841d-e5ed366f84ce nodeName:}" failed. No retries permitted until 2026-09-15 11:39:11.262702092 +0000 UTC m=+38.607810690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls") pod "dns-default-d29g9" (UID: "ddba62f7-548c-494e-841d-e5ed366f84ce") : secret "dns-default-metrics-tls" not found Sep 15 11:39:09.392268 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.392220 2579 generic.go:356] "Generic (PLEG): container finished" podID="e13e3cca-b3e1-4760-be27-3bdbaf1f2512" containerID="575ed198cc8fffe935c4a83514fc0fb786cc0672259b4b31186097b3a0aab331" exitCode=0 Sep 15 11:39:09.392449 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:09.392286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" event={"ID":"e13e3cca-b3e1-4760-be27-3bdbaf1f2512","Type":"ContainerDied","Data":"575ed198cc8fffe935c4a83514fc0fb786cc0672259b4b31186097b3a0aab331"} Sep 15 11:39:10.399232 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:10.398965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" event={"ID":"e13e3cca-b3e1-4760-be27-3bdbaf1f2512","Type":"ContainerStarted","Data":"26f2973d25b31d2120efa48013983ae4dca3728b9772258ea4d48fe7a67806f5"} Sep 15 11:39:10.424022 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:10.423955 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7j5wd" podStartSLOduration=8.957217092 podStartE2EDuration="37.423935681s" totalImagesPullingTime="28.466718589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:10.422296475 +0000 UTC m=+37.767405081" watchObservedRunningTime="2026-09-15 11:39:10.423935681 +0000 UTC m=+37.769044286" Sep 15 11:39:11.277107 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:11.277073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:11.277336 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:11.277191 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:39:11.277336 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:11.277267 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls podName:ddba62f7-548c-494e-841d-e5ed366f84ce nodeName:}" failed. No retries permitted until 2026-09-15 11:39:15.277252489 +0000 UTC m=+42.622361073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls") pod "dns-default-d29g9" (UID: "ddba62f7-548c-494e-841d-e5ed366f84ce") : secret "dns-default-metrics-tls" not found Sep 15 11:39:11.403454 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:11.403422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qr6z5" event={"ID":"84c9ca50-20cf-40e7-a851-37f9e1488015","Type":"ContainerStarted","Data":"6d56edb7ddec2b0015dfdad8e5a6c593971d9d15a3d21dad0983f67d776405cc"} Sep 15 11:39:11.419884 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:11.419833 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qr6z5" podStartSLOduration=1.9344557629999999 podStartE2EDuration="4.419813367s" totalImagesPullingTime="2.485357604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:11.419784338 +0000 UTC m=+38.764892943" watchObservedRunningTime="2026-09-15 11:39:11.419813367 +0000 UTC m=+38.764921973" Sep 15 11:39:15.307102 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:15.307061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:15.307499 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:15.307236 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:39:15.307499 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:15.307300 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls podName:ddba62f7-548c-494e-841d-e5ed366f84ce nodeName:}" failed. No retries permitted until 2026-09-15 11:39:23.307284536 +0000 UTC m=+50.652393119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls") pod "dns-default-d29g9" (UID: "ddba62f7-548c-494e-841d-e5ed366f84ce") : secret "dns-default-metrics-tls" not found Sep 15 11:39:23.369374 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:23.369340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:23.369738 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:23.369481 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:39:23.369738 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:23.369542 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls podName:ddba62f7-548c-494e-841d-e5ed366f84ce nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.369526367 +0000 UTC m=+66.714634968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls") pod "dns-default-d29g9" (UID: "ddba62f7-548c-494e-841d-e5ed366f84ce") : secret "dns-default-metrics-tls" not found Sep 15 11:39:32.162943 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.162906 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hvj6z"] Sep 15 11:39:32.165509 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.165491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.168030 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.168003 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 15 11:39:32.168131 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.168046 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 15 11:39:32.168289 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.168273 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x952v\"" Sep 15 11:39:32.169113 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.169097 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 15 11:39:32.169173 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.169149 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:39:32.171093 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.171074 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq"] Sep 15 11:39:32.173512 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.173489 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:32.173850 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.173831 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 15 11:39:32.175793 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.175776 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 15 11:39:32.175903 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.175803 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 15 11:39:32.175903 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.175807 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 15 11:39:32.176050 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.176037 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-245cf\"" Sep 15 11:39:32.177679 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.177658 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hvj6z"] Sep 15 11:39:32.184985 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.184965 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq"] Sep 15 11:39:32.234110 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.234074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09533bff-aa14-43ed-9121-7783f90a7eef-trusted-ca\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.234245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.234123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09533bff-aa14-43ed-9121-7783f90a7eef-serving-cert\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.234245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.234152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:32.234245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.234193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09533bff-aa14-43ed-9121-7783f90a7eef-config\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.234245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.234241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2x66\" (UniqueName: \"kubernetes.io/projected/de890993-7400-4911-ad83-679ce37e1d63-kube-api-access-m2x66\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:32.234379 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.234275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjq8\" (UniqueName: \"kubernetes.io/projected/09533bff-aa14-43ed-9121-7783f90a7eef-kube-api-access-lsjq8\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.263667 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.263638 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc"] Sep 15 11:39:32.265323 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.265306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.267869 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.267852 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 15 11:39:32.268167 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.268150 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-g2wpl\"" Sep 15 11:39:32.268503 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.268488 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 15 11:39:32.268975 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.268962 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 15 11:39:32.270035 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.270017 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 15 11:39:32.279577 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.279557 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc"] Sep 15 11:39:32.335297 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09533bff-aa14-43ed-9121-7783f90a7eef-serving-cert\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.335297 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.335524 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:32.335524 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:32.335398 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:39:32.335524 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:32.335454 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls podName:de890993-7400-4911-ad83-679ce37e1d63 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:32.83543945 +0000 UTC m=+60.180548034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jpljq" (UID: "de890993-7400-4911-ad83-679ce37e1d63") : secret "samples-operator-tls" not found Sep 15 11:39:32.335524 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6sp\" (UniqueName: \"kubernetes.io/projected/921aeb87-2961-4759-a882-811c53e8c2da-kube-api-access-vf6sp\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.335734 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09533bff-aa14-43ed-9121-7783f90a7eef-config\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.335734 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2x66\" (UniqueName: \"kubernetes.io/projected/de890993-7400-4911-ad83-679ce37e1d63-kube-api-access-m2x66\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:32.335734 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjq8\" (UniqueName: \"kubernetes.io/projected/09533bff-aa14-43ed-9121-7783f90a7eef-kube-api-access-lsjq8\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.335734 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/921aeb87-2961-4759-a882-811c53e8c2da-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.335919 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.335752 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09533bff-aa14-43ed-9121-7783f90a7eef-trusted-ca\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.336238 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.336222 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09533bff-aa14-43ed-9121-7783f90a7eef-config\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.336431 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.336416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09533bff-aa14-43ed-9121-7783f90a7eef-trusted-ca\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.338878 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.338858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/09533bff-aa14-43ed-9121-7783f90a7eef-serving-cert\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.345946 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.345919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2x66\" (UniqueName: \"kubernetes.io/projected/de890993-7400-4911-ad83-679ce37e1d63-kube-api-access-m2x66\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:32.346245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.346226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjq8\" (UniqueName: \"kubernetes.io/projected/09533bff-aa14-43ed-9121-7783f90a7eef-kube-api-access-lsjq8\") pod \"console-operator-696884fb9c-hvj6z\" (UID: \"09533bff-aa14-43ed-9121-7783f90a7eef\") " pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.368630 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.368607 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-mxrhs"] Sep 15 11:39:32.370415 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.370401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.372998 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.372977 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 15 11:39:32.373344 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.373327 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 15 11:39:32.373432 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.373328 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 15 11:39:32.373485 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.373444 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 15 11:39:32.373539 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.373514 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dh5nx\"" Sep 15 11:39:32.382074 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.382050 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 15 11:39:32.383030 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.383008 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-mxrhs"] Sep 15 11:39:32.436467 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/921aeb87-2961-4759-a882-811c53e8c2da-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.436572 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d657879d-7cf6-4d6c-82b8-71fb715059e3-tmp\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.436572 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436514 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d657879d-7cf6-4d6c-82b8-71fb715059e3-service-ca-bundle\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.436572 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kql92\" (UniqueName: \"kubernetes.io/projected/d657879d-7cf6-4d6c-82b8-71fb715059e3-kube-api-access-kql92\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.436688 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.436688 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6sp\" (UniqueName: \"kubernetes.io/projected/921aeb87-2961-4759-a882-811c53e8c2da-kube-api-access-vf6sp\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.436757 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d657879d-7cf6-4d6c-82b8-71fb715059e3-snapshots\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.436757 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:32.436718 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:32.436757 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d657879d-7cf6-4d6c-82b8-71fb715059e3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.436856 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:32.436786 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls podName:921aeb87-2961-4759-a882-811c53e8c2da nodeName:}" failed. No retries permitted until 2026-09-15 11:39:32.93676908 +0000 UTC m=+60.281877681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-wchbc" (UID: "921aeb87-2961-4759-a882-811c53e8c2da") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:32.436856 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.436802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d657879d-7cf6-4d6c-82b8-71fb715059e3-serving-cert\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.437086 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.437068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/921aeb87-2961-4759-a882-811c53e8c2da-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.445385 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.445368 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6sp\" (UniqueName: \"kubernetes.io/projected/921aeb87-2961-4759-a882-811c53e8c2da-kube-api-access-vf6sp\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.475225 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.475192 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:32.538293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.537830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d657879d-7cf6-4d6c-82b8-71fb715059e3-snapshots\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.538293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.537862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d657879d-7cf6-4d6c-82b8-71fb715059e3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.538293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.537895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d657879d-7cf6-4d6c-82b8-71fb715059e3-serving-cert\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.538293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.537923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d657879d-7cf6-4d6c-82b8-71fb715059e3-tmp\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.538293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.537937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d657879d-7cf6-4d6c-82b8-71fb715059e3-service-ca-bundle\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.538293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.537953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kql92\" (UniqueName: \"kubernetes.io/projected/d657879d-7cf6-4d6c-82b8-71fb715059e3-kube-api-access-kql92\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.539091 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.538903 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d657879d-7cf6-4d6c-82b8-71fb715059e3-snapshots\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.539182 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.539151 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d657879d-7cf6-4d6c-82b8-71fb715059e3-tmp\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.539369 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.539346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d657879d-7cf6-4d6c-82b8-71fb715059e3-service-ca-bundle\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.539965 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.539946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d657879d-7cf6-4d6c-82b8-71fb715059e3-trusted-ca-bundle\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.541128 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.541113 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d657879d-7cf6-4d6c-82b8-71fb715059e3-serving-cert\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.547383 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.547358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kql92\" (UniqueName: \"kubernetes.io/projected/d657879d-7cf6-4d6c-82b8-71fb715059e3-kube-api-access-kql92\") pod \"insights-operator-5844b856db-mxrhs\" (UID: \"d657879d-7cf6-4d6c-82b8-71fb715059e3\") " pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.678845 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.678816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-mxrhs" Sep 15 11:39:32.744137 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.744105 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-hvj6z"] Sep 15 11:39:32.748254 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:39:32.748195 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09533bff_aa14_43ed_9121_7783f90a7eef.slice/crio-a8b9b1f597c7f887d5efe45b28df5246299958ad3c71bda197abb47e1578c79c WatchSource:0}: Error finding container a8b9b1f597c7f887d5efe45b28df5246299958ad3c71bda197abb47e1578c79c: Status 404 returned error can't find the container with id a8b9b1f597c7f887d5efe45b28df5246299958ad3c71bda197abb47e1578c79c Sep 15 11:39:32.840519 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.840493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:32.840631 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:32.840618 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:39:32.840688 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:32.840679 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls podName:de890993-7400-4911-ad83-679ce37e1d63 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:33.840660606 +0000 UTC m=+61.185769192 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jpljq" (UID: "de890993-7400-4911-ad83-679ce37e1d63") : secret "samples-operator-tls" not found Sep 15 11:39:32.941150 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.941088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:32.941276 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:32.941243 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:32.941318 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:32.941307 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls podName:921aeb87-2961-4759-a882-811c53e8c2da nodeName:}" failed. No retries permitted until 2026-09-15 11:39:33.941290357 +0000 UTC m=+61.286398954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-wchbc" (UID: "921aeb87-2961-4759-a882-811c53e8c2da") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:32.945671 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:32.945649 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-mxrhs"] Sep 15 11:39:32.948700 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:39:32.948678 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd657879d_7cf6_4d6c_82b8_71fb715059e3.slice/crio-54e6f2c550f6c93fc3e30b39da16dd8af2a98a868f1f701c1e32d8b501bf94aa WatchSource:0}: Error finding container 54e6f2c550f6c93fc3e30b39da16dd8af2a98a868f1f701c1e32d8b501bf94aa: Status 404 returned error can't find the container with id 54e6f2c550f6c93fc3e30b39da16dd8af2a98a868f1f701c1e32d8b501bf94aa Sep 15 11:39:33.377937 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:33.377865 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jhwc4" Sep 15 11:39:33.457241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:33.457181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mxrhs" event={"ID":"d657879d-7cf6-4d6c-82b8-71fb715059e3","Type":"ContainerStarted","Data":"54e6f2c550f6c93fc3e30b39da16dd8af2a98a868f1f701c1e32d8b501bf94aa"} Sep 15 11:39:33.458555 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:33.458517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" event={"ID":"09533bff-aa14-43ed-9121-7783f90a7eef","Type":"ContainerStarted","Data":"a8b9b1f597c7f887d5efe45b28df5246299958ad3c71bda197abb47e1578c79c"} Sep 15 11:39:33.848721 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:33.848684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:33.848884 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:33.848840 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:39:33.848944 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:33.848931 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls podName:de890993-7400-4911-ad83-679ce37e1d63 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:35.848915053 +0000 UTC m=+63.194023637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jpljq" (UID: "de890993-7400-4911-ad83-679ce37e1d63") : secret "samples-operator-tls" not found Sep 15 11:39:33.949650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:33.949611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:33.949826 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:33.949782 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:33.949878 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:33.949870 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls podName:921aeb87-2961-4759-a882-811c53e8c2da nodeName:}" failed. No retries permitted until 2026-09-15 11:39:35.949848059 +0000 UTC m=+63.294956649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-wchbc" (UID: "921aeb87-2961-4759-a882-811c53e8c2da") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:35.465787 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:35.465745 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mxrhs" event={"ID":"d657879d-7cf6-4d6c-82b8-71fb715059e3","Type":"ContainerStarted","Data":"b293cc57766fcd215e4cfc8d50cb982a060069c947746f2dcdad645c9dfaf0b3"} Sep 15 11:39:35.467503 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:35.467481 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/0.log" Sep 15 11:39:35.467620 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:35.467525 2579 generic.go:356] "Generic (PLEG): container finished" podID="09533bff-aa14-43ed-9121-7783f90a7eef" containerID="d33cd2bca04c411c9c95d467979e73f744c926a70bb15a74bf6ffc5dadc18129" exitCode=255 Sep 15 11:39:35.467620 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:35.467552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" event={"ID":"09533bff-aa14-43ed-9121-7783f90a7eef","Type":"ContainerDied","Data":"d33cd2bca04c411c9c95d467979e73f744c926a70bb15a74bf6ffc5dadc18129"} Sep 15 11:39:35.467742 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:35.467729 2579 scope.go:117] "RemoveContainer" containerID="d33cd2bca04c411c9c95d467979e73f744c926a70bb15a74bf6ffc5dadc18129" Sep 15 11:39:35.485898 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:35.485576 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-mxrhs" podStartSLOduration=1.3585503380000001 podStartE2EDuration="3.485555182s" totalImagesPullingTime="2.127004844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:35.484507855 +0000 UTC m=+62.829616462" watchObservedRunningTime="2026-09-15 11:39:35.485555182 +0000 UTC m=+62.830663790" Sep 15 11:39:35.866512 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:35.866415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:35.866649 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:35.866581 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:39:35.866684 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:35.866659 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls podName:de890993-7400-4911-ad83-679ce37e1d63 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.866643035 +0000 UTC m=+67.211751619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jpljq" (UID: "de890993-7400-4911-ad83-679ce37e1d63") : secret "samples-operator-tls" not found Sep 15 11:39:35.967489 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:35.967458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:35.967631 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:35.967572 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:35.967631 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:35.967629 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls podName:921aeb87-2961-4759-a882-811c53e8c2da nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.967616079 +0000 UTC m=+67.312724675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-wchbc" (UID: "921aeb87-2961-4759-a882-811c53e8c2da") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:36.013850 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.013823 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jcrnr"] Sep 15 11:39:36.015681 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.015662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.017982 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.017954 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 15 11:39:36.018085 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.018036 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 15 11:39:36.018085 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.018036 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68t74\"" Sep 15 11:39:36.026116 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.026094 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jcrnr"] Sep 15 11:39:36.068326 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.068294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fda5b32-6097-4f9b-8a7d-df00f6add96a-crio-socket\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.068326 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.068326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.068513 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.068373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fda5b32-6097-4f9b-8a7d-df00f6add96a-data-volume\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.068513 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.068389 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.068513 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.068435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vwp\" (UniqueName: \"kubernetes.io/projected/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-api-access-89vwp\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.169370 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.169301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fda5b32-6097-4f9b-8a7d-df00f6add96a-data-volume\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.169370 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.169333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.169370 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.169349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vwp\" (UniqueName: \"kubernetes.io/projected/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-api-access-89vwp\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.169537 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.169421 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 11:39:36.169577 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.169556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fda5b32-6097-4f9b-8a7d-df00f6add96a-crio-socket\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.169609 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.169592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.169655 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.169611 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fda5b32-6097-4f9b-8a7d-df00f6add96a-data-volume\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.169655 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.169646 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:36.669616341 +0000 UTC m=+64.014724938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : configmap "kube-rbac-proxy" not found Sep 15 11:39:36.169756 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.169695 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:39:36.169756 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.169699 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fda5b32-6097-4f9b-8a7d-df00f6add96a-crio-socket\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.169756 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.169739 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:36.669720285 +0000 UTC m=+64.014828870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : secret "insights-runtime-extractor-tls" not found Sep 15 11:39:36.182418 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.182394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vwp\" (UniqueName: \"kubernetes.io/projected/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-api-access-89vwp\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.471816 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.471791 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/1.log" Sep 15 11:39:36.472245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.472150 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/0.log" Sep 15 11:39:36.472245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.472183 2579 generic.go:356] "Generic (PLEG): container finished" podID="09533bff-aa14-43ed-9121-7783f90a7eef" containerID="0c6cf973a0077fcc086db679826618179c336e4b41d6ced182355da6179d72e9" exitCode=255 Sep 15 11:39:36.472245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.472232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" event={"ID":"09533bff-aa14-43ed-9121-7783f90a7eef","Type":"ContainerDied","Data":"0c6cf973a0077fcc086db679826618179c336e4b41d6ced182355da6179d72e9"} Sep 15 11:39:36.472352 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.472312 2579 scope.go:117] "RemoveContainer" containerID="d33cd2bca04c411c9c95d467979e73f744c926a70bb15a74bf6ffc5dadc18129" Sep 15 11:39:36.472545 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.472526 2579 scope.go:117] "RemoveContainer" containerID="0c6cf973a0077fcc086db679826618179c336e4b41d6ced182355da6179d72e9" Sep 15 11:39:36.472701 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.472684 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hvj6z_openshift-console-operator(09533bff-aa14-43ed-9121-7783f90a7eef)\"" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" podUID="09533bff-aa14-43ed-9121-7783f90a7eef" Sep 15 11:39:36.674281 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.674252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.674417 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:36.674306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:36.674417 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.674384 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:39:36.674417 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.674413 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 11:39:36.674559 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.674447 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:37.674432087 +0000 UTC m=+65.019540674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : secret "insights-runtime-extractor-tls" not found Sep 15 11:39:36.674559 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:36.674461 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:37.674454079 +0000 UTC m=+65.019562663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : configmap "kube-rbac-proxy" not found Sep 15 11:39:37.478323 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:37.478294 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/1.log" Sep 15 11:39:37.478688 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:37.478611 2579 scope.go:117] "RemoveContainer" containerID="0c6cf973a0077fcc086db679826618179c336e4b41d6ced182355da6179d72e9" Sep 15 11:39:37.478767 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:37.478749 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hvj6z_openshift-console-operator(09533bff-aa14-43ed-9121-7783f90a7eef)\"" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" podUID="09533bff-aa14-43ed-9121-7783f90a7eef" Sep 15 11:39:37.681558 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:37.681516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:37.681752 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:37.681646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:37.681752 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:37.681653 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 11:39:37.681752 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:37.681733 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.681711933 +0000 UTC m=+67.026820534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : configmap "kube-rbac-proxy" not found Sep 15 11:39:37.681904 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:37.681776 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:39:37.681904 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:37.681817 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:39.681805531 +0000 UTC m=+67.026914118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : secret "insights-runtime-extractor-tls" not found Sep 15 11:39:37.758479 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:37.758399 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2kk9v_99912355-5a7e-4bae-9df2-d1a431b0b5fe/dns-node-resolver/0.log" Sep 15 11:39:38.757394 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.757364 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-69tt7_840cfbe9-71a6-4d47-b285-292f5462bb62/node-ca/0.log" Sep 15 11:39:38.959780 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.959743 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq"] Sep 15 11:39:38.962885 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.962868 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" Sep 15 11:39:38.964985 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.964959 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m4zpk\"" Sep 15 11:39:38.965114 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.964959 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 15 11:39:38.965680 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.965664 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 15 11:39:38.973564 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.973540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq"] Sep 15 11:39:38.993075 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.993039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8p2z\" (UniqueName: \"kubernetes.io/projected/bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf-kube-api-access-n8p2z\") pod \"migrator-8b4d697c5-vqnhq\" (UID: \"bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" Sep 15 11:39:38.993297 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.993089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:39:38.993297 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.993125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:38.995507 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.995485 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 15 11:39:38.995601 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:38.995579 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 15 11:39:39.003403 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.003374 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 15 11:39:39.003504 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.003460 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs podName:0e68ffcd-641e-4bbf-9703-5397e737b4b6 nodeName:}" failed. No retries permitted until 2026-09-15 11:40:43.003442359 +0000 UTC m=+130.348550947 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs") pod "network-metrics-daemon-q7vl8" (UID: "0e68ffcd-641e-4bbf-9703-5397e737b4b6") : secret "metrics-daemon-secret" not found Sep 15 11:39:39.005682 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.005659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/72de5f15-c37b-4c5b-ab3b-e182a42b06ab-original-pull-secret\") pod \"global-pull-secret-syncer-p469s\" (UID: \"72de5f15-c37b-4c5b-ab3b-e182a42b06ab\") " pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:39.091522 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.091429 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p469s" Sep 15 11:39:39.094351 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.094332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:39.094402 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.094361 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8p2z\" (UniqueName: \"kubernetes.io/projected/bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf-kube-api-access-n8p2z\") pod \"migrator-8b4d697c5-vqnhq\" (UID: \"bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" Sep 15 11:39:39.097058 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.097038 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 15 11:39:39.107256 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.107231 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 15 11:39:39.109836 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.109812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8p2z\" (UniqueName: \"kubernetes.io/projected/bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf-kube-api-access-n8p2z\") pod \"migrator-8b4d697c5-vqnhq\" (UID: \"bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" Sep 15 11:39:39.117989 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.117959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kx7\" (UniqueName: \"kubernetes.io/projected/6ac50613-e1b5-4278-9430-e64618567054-kube-api-access-62kx7\") pod \"network-check-target-4ztt5\" (UID: \"6ac50613-e1b5-4278-9430-e64618567054\") " pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:39.198688 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.198653 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bwddj\"" Sep 15 11:39:39.206489 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.206454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:39.271606 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.271571 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" Sep 15 11:39:39.365579 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.363550 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p469s"] Sep 15 11:39:39.368522 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:39:39.368494 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72de5f15_c37b_4c5b_ab3b_e182a42b06ab.slice/crio-1e21cff4c0ee55a733c21d33f522b43cb9481d1044a5c0a659f68a65fd3ca067 WatchSource:0}: Error finding container 1e21cff4c0ee55a733c21d33f522b43cb9481d1044a5c0a659f68a65fd3ca067: Status 404 returned error can't find the container with id 1e21cff4c0ee55a733c21d33f522b43cb9481d1044a5c0a659f68a65fd3ca067 Sep 15 11:39:39.396383 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.396344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:39:39.396547 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.396512 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 15 11:39:39.396602 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.396591 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls podName:ddba62f7-548c-494e-841d-e5ed366f84ce nodeName:}" failed. No retries permitted until 2026-09-15 11:40:11.396575334 +0000 UTC m=+98.741683917 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls") pod "dns-default-d29g9" (UID: "ddba62f7-548c-494e-841d-e5ed366f84ce") : secret "dns-default-metrics-tls" not found Sep 15 11:39:39.479080 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.479046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4ztt5"] Sep 15 11:39:39.482014 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:39:39.481977 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac50613_e1b5_4278_9430_e64618567054.slice/crio-2ce233d50841866877eb733e22dea6c3d675710ef7e695e981170e370e780114 WatchSource:0}: Error finding container 2ce233d50841866877eb733e22dea6c3d675710ef7e695e981170e370e780114: Status 404 returned error can't find the container with id 2ce233d50841866877eb733e22dea6c3d675710ef7e695e981170e370e780114 Sep 15 11:39:39.484940 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.484909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p469s" event={"ID":"72de5f15-c37b-4c5b-ab3b-e182a42b06ab","Type":"ContainerStarted","Data":"1e21cff4c0ee55a733c21d33f522b43cb9481d1044a5c0a659f68a65fd3ca067"} Sep 15 11:39:39.545127 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.545092 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq"] Sep 15 11:39:39.547838 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:39:39.547808 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6d16c9_e568_4b2c_910c_c78a6ba8e3cf.slice/crio-2852f2536f9bee91d1c314764cab3b194e765ee72a0f893a294a29013591fae7 WatchSource:0}: Error finding container 2852f2536f9bee91d1c314764cab3b194e765ee72a0f893a294a29013591fae7: Status 404 returned error can't find the container with id 2852f2536f9bee91d1c314764cab3b194e765ee72a0f893a294a29013591fae7 Sep 15 11:39:39.698422 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.698378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:39.698629 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.698434 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:39:39.698629 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.698461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:39.698629 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.698515 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:43.698497427 +0000 UTC m=+71.043606012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : secret "insights-runtime-extractor-tls" not found Sep 15 11:39:39.698629 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.698530 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 11:39:39.698629 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.698573 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:43.6985624 +0000 UTC m=+71.043670984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : configmap "kube-rbac-proxy" not found Sep 15 11:39:39.900889 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.900854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:39.901311 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.900995 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 15 11:39:39.901311 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:39.901063 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls podName:de890993-7400-4911-ad83-679ce37e1d63 nodeName:}" failed. No retries permitted until 2026-09-15 11:39:47.901048263 +0000 UTC m=+75.246156851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-jpljq" (UID: "de890993-7400-4911-ad83-679ce37e1d63") : secret "samples-operator-tls" not found Sep 15 11:39:39.957862 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:39.957783 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qr6z5_84c9ca50-20cf-40e7-a851-37f9e1488015/serve-healthcheck-canary/0.log" Sep 15 11:39:40.002381 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:40.002328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:40.002574 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:40.002464 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:40.002574 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:40.002543 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls podName:921aeb87-2961-4759-a882-811c53e8c2da nodeName:}" failed. No retries permitted until 2026-09-15 11:39:48.002522225 +0000 UTC m=+75.347630812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-wchbc" (UID: "921aeb87-2961-4759-a882-811c53e8c2da") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:40.490889 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:40.490849 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" event={"ID":"bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf","Type":"ContainerStarted","Data":"2852f2536f9bee91d1c314764cab3b194e765ee72a0f893a294a29013591fae7"} Sep 15 11:39:40.492451 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:40.492390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4ztt5" event={"ID":"6ac50613-e1b5-4278-9430-e64618567054","Type":"ContainerStarted","Data":"2ce233d50841866877eb733e22dea6c3d675710ef7e695e981170e370e780114"} Sep 15 11:39:41.498292 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:41.498246 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" event={"ID":"bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf","Type":"ContainerStarted","Data":"4086a96793462d415855804ddeed7cbdb1b6b92617515b6867a7ee34f803cff8"} Sep 15 11:39:41.498731 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:41.498312 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" event={"ID":"bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf","Type":"ContainerStarted","Data":"62102a371dfad2f82d6ec88182091b3c3dd323ee55f66ef773d97fe38750a9f5"} Sep 15 11:39:41.517001 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:41.516949 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vqnhq" podStartSLOduration=2.237192698 podStartE2EDuration="3.516931856s" totalImagesPullingTime="1.279739158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:41.515604588 +0000 UTC m=+68.860713209" watchObservedRunningTime="2026-09-15 11:39:41.516931856 +0000 UTC m=+68.862040487" Sep 15 11:39:42.475531 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:42.475490 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:42.475724 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:42.475561 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:39:42.475974 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:42.475960 2579 scope.go:117] "RemoveContainer" containerID="0c6cf973a0077fcc086db679826618179c336e4b41d6ced182355da6179d72e9" Sep 15 11:39:42.476148 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:42.476128 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-hvj6z_openshift-console-operator(09533bff-aa14-43ed-9121-7783f90a7eef)\"" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" podUID="09533bff-aa14-43ed-9121-7783f90a7eef" Sep 15 11:39:43.736533 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:43.736500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:43.736979 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:43.736566 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:43.736979 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:43.736680 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:39:43.736979 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:43.736721 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 15 11:39:43.736979 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:43.736749 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:51.736732888 +0000 UTC m=+79.081841475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : secret "insights-runtime-extractor-tls" not found Sep 15 11:39:43.736979 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:43.736786 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:39:51.736771967 +0000 UTC m=+79.081880566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : configmap "kube-rbac-proxy" not found Sep 15 11:39:44.507648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:44.507611 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4ztt5" event={"ID":"6ac50613-e1b5-4278-9430-e64618567054","Type":"ContainerStarted","Data":"8a0f4a318bf75749cf44a42f3b5f5ac76420d8d0ea42e9128d260fad848bac4f"} Sep 15 11:39:44.507849 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:44.507703 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:39:44.509081 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:44.509057 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p469s" event={"ID":"72de5f15-c37b-4c5b-ab3b-e182a42b06ab","Type":"ContainerStarted","Data":"7ad1317127069d79673330e411ce9237f3525fac74301245815a9a6107f0495f"} Sep 15 11:39:44.525310 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:44.525256 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4ztt5" podStartSLOduration=66.975353899 podStartE2EDuration="1m11.525239838s" totalImagesPullingTime="4.549885939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:44.524947565 +0000 UTC m=+71.870056173" watchObservedRunningTime="2026-09-15 11:39:44.525239838 +0000 UTC m=+71.870348446" Sep 15 11:39:44.540692 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:44.540644 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p469s" podStartSLOduration=66.872273965 podStartE2EDuration="1m11.540626067s" totalImagesPullingTime="4.668352102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:44.540175871 +0000 UTC m=+71.885284476" watchObservedRunningTime="2026-09-15 11:39:44.540626067 +0000 UTC m=+71.885734672" Sep 15 11:39:47.968880 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:47.968836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:47.971423 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:47.971397 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/de890993-7400-4911-ad83-679ce37e1d63-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-jpljq\" (UID: \"de890993-7400-4911-ad83-679ce37e1d63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:48.069832 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:48.069790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:39:48.070015 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:48.069920 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:48.070015 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:48.069978 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls podName:921aeb87-2961-4759-a882-811c53e8c2da nodeName:}" failed. No retries permitted until 2026-09-15 11:40:04.069963826 +0000 UTC m=+91.415072409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-wchbc" (UID: "921aeb87-2961-4759-a882-811c53e8c2da") : secret "cluster-monitoring-operator-tls" not found Sep 15 11:39:48.085361 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:48.085330 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-245cf\"" Sep 15 11:39:48.093401 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:48.093373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" Sep 15 11:39:48.364393 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:48.364305 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq"] Sep 15 11:39:48.521999 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:48.521965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" event={"ID":"de890993-7400-4911-ad83-679ce37e1d63","Type":"ContainerStarted","Data":"a4a8ebb0113fbe1beff06336c47ec818814de4f302793103a3f81b4df50876d9"} Sep 15 11:39:50.530582 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:50.530492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" event={"ID":"de890993-7400-4911-ad83-679ce37e1d63","Type":"ContainerStarted","Data":"5024ea3074a42ceea1d8c1ddc872261166ce376741edf5a62c74b949bf3fd2ed"} Sep 15 11:39:50.530582 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:50.530554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" event={"ID":"de890993-7400-4911-ad83-679ce37e1d63","Type":"ContainerStarted","Data":"d096d3082c6918833bbcaeaf3930a01c6f993b313cf5edb3a3857cef6d25aea9"} Sep 15 11:39:50.549705 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:50.549644 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-jpljq" podStartSLOduration=16.690544023 podStartE2EDuration="18.549626737s" totalImagesPullingTime="1.859082714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:50.548844267 +0000 UTC m=+77.893952873" watchObservedRunningTime="2026-09-15 11:39:50.549626737 +0000 UTC m=+77.894735342" Sep 15 11:39:51.797374 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:51.797338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:51.797770 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:51.797417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:51.797770 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:51.797515 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 15 11:39:51.797770 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:51.797574 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls podName:1fda5b32-6097-4f9b-8a7d-df00f6add96a nodeName:}" failed. No retries permitted until 2026-09-15 11:40:07.797561721 +0000 UTC m=+95.142670304 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jcrnr" (UID: "1fda5b32-6097-4f9b-8a7d-df00f6add96a") : secret "insights-runtime-extractor-tls" not found Sep 15 11:39:51.797878 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:51.797804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fda5b32-6097-4f9b-8a7d-df00f6add96a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:39:55.182171 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:55.182066 2579 scope.go:117] "RemoveContainer" containerID="0c6cf973a0077fcc086db679826618179c336e4b41d6ced182355da6179d72e9" Sep 15 11:39:55.548161 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:55.548087 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:39:55.548563 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:55.548547 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/1.log" Sep 15 11:39:55.548626 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:55.548581 2579 generic.go:356] "Generic (PLEG): container finished" podID="09533bff-aa14-43ed-9121-7783f90a7eef" containerID="b5bc54c12c4af27ae2c4efbbdb4366e2b63b0cebc535b542a0e66dc06d83e4f8" exitCode=255 Sep 15 11:39:55.548626 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:55.548612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" event={"ID":"09533bff-aa14-43ed-9121-7783f90a7eef","Type":"ContainerDied","Data":"b5bc54c12c4af27ae2c4efbbdb4366e2b63b0cebc535b542a0e66dc06d83e4f8"} Sep 15 11:39:55.548694 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:55.548656 2579 scope.go:117] "RemoveContainer" containerID="0c6cf973a0077fcc086db679826618179c336e4b41d6ced182355da6179d72e9" Sep 15 11:39:55.548940 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:55.548920 2579 scope.go:117] "RemoveContainer" containerID="b5bc54c12c4af27ae2c4efbbdb4366e2b63b0cebc535b542a0e66dc06d83e4f8" Sep 15 11:39:55.549101 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:39:55.549082 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-hvj6z_openshift-console-operator(09533bff-aa14-43ed-9121-7783f90a7eef)\"" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" podUID="09533bff-aa14-43ed-9121-7783f90a7eef" Sep 15 11:39:56.553414 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:39:56.553380 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:40:02.475603 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:02.475562 2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:40:02.476122 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:02.475629 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:40:02.476122 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:02.476012 2579 scope.go:117] "RemoveContainer" containerID="b5bc54c12c4af27ae2c4efbbdb4366e2b63b0cebc535b542a0e66dc06d83e4f8" Sep 15 11:40:02.476274 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:40:02.476252 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-hvj6z_openshift-console-operator(09533bff-aa14-43ed-9121-7783f90a7eef)\"" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" podUID="09533bff-aa14-43ed-9121-7783f90a7eef" Sep 15 11:40:03.075795 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.075761 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq"] Sep 15 11:40:03.080147 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.080127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.082744 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.082727 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 15 11:40:03.083564 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.083545 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 15 11:40:03.083689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.083582 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 15 11:40:03.083689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.083590 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 15 11:40:03.083689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.083606 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 15 11:40:03.083689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.083611 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 15 11:40:03.083689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.083680 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 15 11:40:03.091928 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.091906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq"] Sep 15 11:40:03.181122 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.181092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d80778-de1a-4f86-bf66-c6d1e381877a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.181122 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.181123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-hub\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.181337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.181156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.181337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.181226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.181337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.181284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkxj\" (UniqueName: \"kubernetes.io/projected/24d80778-de1a-4f86-bf66-c6d1e381877a-kube-api-access-4gkxj\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.181337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.181320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-ca\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.281855 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.281819 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkxj\" (UniqueName: \"kubernetes.io/projected/24d80778-de1a-4f86-bf66-c6d1e381877a-kube-api-access-4gkxj\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.281855 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.281869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-ca\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.282099 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.282059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d80778-de1a-4f86-bf66-c6d1e381877a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.282099 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.282095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-hub\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.282239 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.282138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.282239 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.282170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.282797 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.282772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/24d80778-de1a-4f86-bf66-c6d1e381877a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.284523 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.284492 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-ca\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.285094 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.285069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.285171 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.285111 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-hub\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.285308 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.285286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24d80778-de1a-4f86-bf66-c6d1e381877a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.291914 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.291895 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkxj\" (UniqueName: \"kubernetes.io/projected/24d80778-de1a-4f86-bf66-c6d1e381877a-kube-api-access-4gkxj\") pod \"cluster-proxy-proxy-agent-5df9c5d585-8rvxq\" (UID: \"24d80778-de1a-4f86-bf66-c6d1e381877a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.398238 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.398151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" Sep 15 11:40:03.668739 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:03.668653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq"] Sep 15 11:40:03.672453 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:40:03.672426 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24d80778_de1a_4f86_bf66_c6d1e381877a.slice/crio-bc599462d03a89643e1a487fdaaa4202c1cf2602be9e470cefa1d813e8f01b6b WatchSource:0}: Error finding container bc599462d03a89643e1a487fdaaa4202c1cf2602be9e470cefa1d813e8f01b6b: Status 404 returned error can't find the container with id bc599462d03a89643e1a487fdaaa4202c1cf2602be9e470cefa1d813e8f01b6b Sep 15 11:40:04.089150 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:04.089120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:40:04.091620 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:04.091600 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/921aeb87-2961-4759-a882-811c53e8c2da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-wchbc\" (UID: \"921aeb87-2961-4759-a882-811c53e8c2da\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:40:04.375960 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:04.375881 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-g2wpl\"" Sep 15 11:40:04.384269 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:04.384248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" Sep 15 11:40:04.579386 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:04.579354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" event={"ID":"24d80778-de1a-4f86-bf66-c6d1e381877a","Type":"ContainerStarted","Data":"bc599462d03a89643e1a487fdaaa4202c1cf2602be9e470cefa1d813e8f01b6b"} Sep 15 11:40:04.656744 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:04.656671 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc"] Sep 15 11:40:04.659887 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:40:04.659862 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921aeb87_2961_4759_a882_811c53e8c2da.slice/crio-eb64afbb9184486ca705cbb3ae71dd5891982c470b4de0e8049376e17544caed WatchSource:0}: Error finding container eb64afbb9184486ca705cbb3ae71dd5891982c470b4de0e8049376e17544caed: Status 404 returned error can't find the container with id eb64afbb9184486ca705cbb3ae71dd5891982c470b4de0e8049376e17544caed Sep 15 11:40:05.583801 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:05.583773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" event={"ID":"921aeb87-2961-4759-a882-811c53e8c2da","Type":"ContainerStarted","Data":"eb64afbb9184486ca705cbb3ae71dd5891982c470b4de0e8049376e17544caed"} Sep 15 11:40:07.591531 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:07.591492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" event={"ID":"921aeb87-2961-4759-a882-811c53e8c2da","Type":"ContainerStarted","Data":"3c98278b71409874fd7106ce9f83fd93d0ae4b600f297aa5542ef928dcca624c"} Sep 15 11:40:07.607908 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:07.607850 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-wchbc" podStartSLOduration=33.438133521 podStartE2EDuration="35.607831617s" totalImagesPullingTime="2.169698096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:07.60739435 +0000 UTC m=+94.952502982" watchObservedRunningTime="2026-09-15 11:40:07.607831617 +0000 UTC m=+94.952940225" Sep 15 11:40:07.819772 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:07.819736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:40:07.822144 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:07.822118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fda5b32-6097-4f9b-8a7d-df00f6add96a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jcrnr\" (UID: \"1fda5b32-6097-4f9b-8a7d-df00f6add96a\") " pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:40:07.824122 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:07.824107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jcrnr" Sep 15 11:40:08.097533 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:08.097497 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jcrnr"] Sep 15 11:40:08.101307 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:40:08.101266 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fda5b32_6097_4f9b_8a7d_df00f6add96a.slice/crio-48648c224006ab849cdeaee3d44fb22a58783201bd298747ecb4e3e052a60f50 WatchSource:0}: Error finding container 48648c224006ab849cdeaee3d44fb22a58783201bd298747ecb4e3e052a60f50: Status 404 returned error can't find the container with id 48648c224006ab849cdeaee3d44fb22a58783201bd298747ecb4e3e052a60f50 Sep 15 11:40:08.597643 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:08.597598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jcrnr" event={"ID":"1fda5b32-6097-4f9b-8a7d-df00f6add96a","Type":"ContainerStarted","Data":"0df707f24d4de071d67530f82ff109706763c7249c9668e95d7ecff0159d4df5"} Sep 15 11:40:08.598087 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:08.597710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jcrnr" event={"ID":"1fda5b32-6097-4f9b-8a7d-df00f6add96a","Type":"ContainerStarted","Data":"48648c224006ab849cdeaee3d44fb22a58783201bd298747ecb4e3e052a60f50"} Sep 15 11:40:11.448995 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.448956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:40:11.451408 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.451380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddba62f7-548c-494e-841d-e5ed366f84ce-metrics-tls\") pod \"dns-default-d29g9\" (UID: \"ddba62f7-548c-494e-841d-e5ed366f84ce\") " pod="openshift-dns/dns-default-d29g9" Sep 15 11:40:11.609650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.609561 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" event={"ID":"24d80778-de1a-4f86-bf66-c6d1e381877a","Type":"ContainerStarted","Data":"4e57f5aa634cb418446eb615bfd043b419c97131eb5da028991aaa36a2221c48"} Sep 15 11:40:11.609650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.609608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" event={"ID":"24d80778-de1a-4f86-bf66-c6d1e381877a","Type":"ContainerStarted","Data":"bb169cd2686c282ce9a80b357a4a89d3ad1b19159e8e96bbead832d1b9d90270"} Sep 15 11:40:11.609650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.609624 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" event={"ID":"24d80778-de1a-4f86-bf66-c6d1e381877a","Type":"ContainerStarted","Data":"786f54006dcf7593a2ef35ca44dd07782d060db832013e5e170d2574764678ac"} Sep 15 11:40:11.611432 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.611410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jcrnr" event={"ID":"1fda5b32-6097-4f9b-8a7d-df00f6add96a","Type":"ContainerStarted","Data":"bb57c5ee27753d38a49d137d5ebd2fa9b0f6280fb0fd5e9f9996b0ab594e42fe"} Sep 15 11:40:11.627509 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.627458 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5df9c5d585-8rvxq" podStartSLOduration=1.423488404 podStartE2EDuration="8.627443504s" totalImagesPullingTime="7.2039551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:11.627105647 +0000 UTC m=+98.972214252" watchObservedRunningTime="2026-09-15 11:40:11.627443504 +0000 UTC m=+98.972552110" Sep 15 11:40:11.739314 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.739280 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hx5hb\"" Sep 15 11:40:11.748118 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:11.748086 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d29g9" Sep 15 11:40:12.028007 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:12.027974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d29g9"] Sep 15 11:40:12.032255 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:40:12.032220 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddba62f7_548c_494e_841d_e5ed366f84ce.slice/crio-7fb568c9d732587a1902d6ffb524710d15bf973e0ebbcc96d2fe774938b8b90a WatchSource:0}: Error finding container 7fb568c9d732587a1902d6ffb524710d15bf973e0ebbcc96d2fe774938b8b90a: Status 404 returned error can't find the container with id 7fb568c9d732587a1902d6ffb524710d15bf973e0ebbcc96d2fe774938b8b90a Sep 15 11:40:12.618049 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:12.618013 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jcrnr" event={"ID":"1fda5b32-6097-4f9b-8a7d-df00f6add96a","Type":"ContainerStarted","Data":"42917e8a1a9092df8f99c89f16c6aa0d45d285900f048f4d3e6544aad0cba238"} Sep 15 11:40:12.619327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:12.619303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d29g9" event={"ID":"ddba62f7-548c-494e-841d-e5ed366f84ce","Type":"ContainerStarted","Data":"7fb568c9d732587a1902d6ffb524710d15bf973e0ebbcc96d2fe774938b8b90a"} Sep 15 11:40:12.637888 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:12.637818 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jcrnr" podStartSLOduration=33.356025658 podStartE2EDuration="37.63779764s" totalImagesPullingTime="4.281771982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:39:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:12.635372941 +0000 UTC m=+99.980481560" watchObservedRunningTime="2026-09-15 11:40:12.63779764 +0000 UTC m=+99.982906249" Sep 15 11:40:14.627972 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.627938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d29g9" event={"ID":"ddba62f7-548c-494e-841d-e5ed366f84ce","Type":"ContainerStarted","Data":"2dcf85cd36bc6ab147dfa00b2662c1ec36abd2594f76a46923a5837c8e5116fb"} Sep 15 11:40:14.628421 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.627986 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d29g9" event={"ID":"ddba62f7-548c-494e-841d-e5ed366f84ce","Type":"ContainerStarted","Data":"4c6ee279b93c0ed55519623dee8c9bddb27c737cb0e671d494ee52369508006b"} Sep 15 11:40:14.628421 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.628094 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d29g9" Sep 15 11:40:14.648092 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.648030 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d29g9" podStartSLOduration=66.049079443 podStartE2EDuration="1m7.648014377s" totalImagesPullingTime="1.598934934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:14.647574752 +0000 UTC m=+101.992683358" watchObservedRunningTime="2026-09-15 11:40:14.648014377 +0000 UTC m=+101.993122982" Sep 15 11:40:14.779962 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.779929 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j2gvq"] Sep 15 11:40:14.783065 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.783047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.785709 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.785689 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bk6sw\"" Sep 15 11:40:14.785813 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.785688 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 15 11:40:14.785813 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.785724 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 15 11:40:14.785813 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.785761 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 15 11:40:14.785956 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.785762 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 15 11:40:14.879098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-root\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.879098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.879320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqm2k\" (UniqueName: \"kubernetes.io/projected/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-kube-api-access-vqm2k\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.879320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-tls\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.879320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-textfile\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.879320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879252 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.879320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-metrics-client-ca\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.879320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-wtmp\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.879320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.879304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-sys\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980577 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980577 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-metrics-client-ca\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980807 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980609 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-wtmp\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980807 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-sys\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980807 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980747 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-wtmp\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-root\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-sys\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqm2k\" (UniqueName: \"kubernetes.io/projected/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-kube-api-access-vqm2k\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-root\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.980961 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.980958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-tls\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.981301 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.981002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-textfile\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.981301 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:40:14.981085 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 15 11:40:14.981301 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:40:14.981147 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-tls podName:70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9 nodeName:}" failed. No retries permitted until 2026-09-15 11:40:15.481126915 +0000 UTC m=+102.826235500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-tls") pod "node-exporter-j2gvq" (UID: "70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9") : secret "node-exporter-tls" not found Sep 15 11:40:14.981301 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.981271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-metrics-client-ca\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.981461 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.981380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-textfile\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.981461 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.981407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.983136 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.983112 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:14.991831 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:14.991810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqm2k\" (UniqueName: \"kubernetes.io/projected/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-kube-api-access-vqm2k\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:15.485221 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:15.485163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-tls\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:15.487557 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:15.487527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9-node-exporter-tls\") pod \"node-exporter-j2gvq\" (UID: \"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9\") " pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:15.514764 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:15.514733 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4ztt5" Sep 15 11:40:15.692455 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:15.692415 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2gvq" Sep 15 11:40:15.707256 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:40:15.707222 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70178ee2_378c_4dbd_bdf6_7d45a5d4a7e9.slice/crio-743afffc9c318f98c9464e2fc1fde4ea5b9d90bdacc169c1c700b7416efdfe96 WatchSource:0}: Error finding container 743afffc9c318f98c9464e2fc1fde4ea5b9d90bdacc169c1c700b7416efdfe96: Status 404 returned error can't find the container with id 743afffc9c318f98c9464e2fc1fde4ea5b9d90bdacc169c1c700b7416efdfe96 Sep 15 11:40:16.636346 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:16.636307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2gvq" event={"ID":"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9","Type":"ContainerStarted","Data":"743afffc9c318f98c9464e2fc1fde4ea5b9d90bdacc169c1c700b7416efdfe96"} Sep 15 11:40:17.182252 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:17.182219 2579 scope.go:117] "RemoveContainer" containerID="b5bc54c12c4af27ae2c4efbbdb4366e2b63b0cebc535b542a0e66dc06d83e4f8" Sep 15 11:40:17.641094 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:17.641061 2579 generic.go:356] "Generic (PLEG): container finished" podID="70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9" containerID="c80e6addc98e137001ce7c5674cfb2c521c533031b4b43fc5386fbbb002e3533" exitCode=0 Sep 15 11:40:17.641356 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:17.641149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2gvq" event={"ID":"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9","Type":"ContainerDied","Data":"c80e6addc98e137001ce7c5674cfb2c521c533031b4b43fc5386fbbb002e3533"} Sep 15 11:40:17.643240 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:17.643192 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:40:17.643337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:17.643275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" event={"ID":"09533bff-aa14-43ed-9121-7783f90a7eef","Type":"ContainerStarted","Data":"060d4bdd6d223cb98348b699cd23e60088a0725637229e54e194ece732431acb"} Sep 15 11:40:17.643559 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:17.643536 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:40:17.682149 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:17.682100 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" podStartSLOduration=43.356328433 podStartE2EDuration="45.682081762s" totalImagesPullingTime="2.325753329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:17.681062509 +0000 UTC m=+105.026171111" watchObservedRunningTime="2026-09-15 11:40:17.682081762 +0000 UTC m=+105.027190368" Sep 15 11:40:17.860868 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:17.860843 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-hvj6z" Sep 15 11:40:18.648739 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:18.648695 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2gvq" event={"ID":"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9","Type":"ContainerStarted","Data":"bb001beedefa215953ef02a734bff374202c3abb6728f9538fb86d0156043d07"} Sep 15 11:40:18.649238 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:18.648752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2gvq" event={"ID":"70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9","Type":"ContainerStarted","Data":"2bff9b8633253f309beddc8ce6f14df90f2ab0f1b2da0a5fc8aefad1ebea99c9"} Sep 15 11:40:18.669678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:18.669625 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j2gvq" podStartSLOduration=3.019371266 podStartE2EDuration="4.669607723s" totalImagesPullingTime="1.567882754s" totalInitContainerRuntime="82.353703ms" isStatefulPod=true podCreationTimestamp="2026-09-15 11:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:18.668411879 +0000 UTC m=+106.013520485" watchObservedRunningTime="2026-09-15 11:40:18.669607723 +0000 UTC m=+106.014716328" Sep 15 11:40:24.633657 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:24.633630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d29g9" Sep 15 11:40:43.100238 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:43.100180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:40:43.102736 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:43.102709 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e68ffcd-641e-4bbf-9703-5397e737b4b6-metrics-certs\") pod \"network-metrics-daemon-q7vl8\" (UID: \"0e68ffcd-641e-4bbf-9703-5397e737b4b6\") " pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:40:43.404176 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:43.404107 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmnh2\"" Sep 15 11:40:43.412529 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:43.412507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q7vl8" Sep 15 11:40:43.739365 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:43.739340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q7vl8"] Sep 15 11:40:43.742001 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:40:43.741975 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e68ffcd_641e_4bbf_9703_5397e737b4b6.slice/crio-c48e04988ffa17c0eecd28923104c652a50290d85825c8a4aa9d5e471b8c5178 WatchSource:0}: Error finding container c48e04988ffa17c0eecd28923104c652a50290d85825c8a4aa9d5e471b8c5178: Status 404 returned error can't find the container with id c48e04988ffa17c0eecd28923104c652a50290d85825c8a4aa9d5e471b8c5178 Sep 15 11:40:44.737369 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:44.737334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7vl8" event={"ID":"0e68ffcd-641e-4bbf-9703-5397e737b4b6","Type":"ContainerStarted","Data":"c48e04988ffa17c0eecd28923104c652a50290d85825c8a4aa9d5e471b8c5178"} Sep 15 11:40:45.742906 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:45.742865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7vl8" event={"ID":"0e68ffcd-641e-4bbf-9703-5397e737b4b6","Type":"ContainerStarted","Data":"4e8bd34d8a09c2b0f25d991c3810539326c91e28ee33ed8cbdfacd2dc035355e"} Sep 15 11:40:45.743297 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:45.742917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q7vl8" event={"ID":"0e68ffcd-641e-4bbf-9703-5397e737b4b6","Type":"ContainerStarted","Data":"de251719f4c427c477522c24e9fe3e639e2931dc43868aa695903431cd82dd77"} Sep 15 11:40:45.762092 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:40:45.762043 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q7vl8" podStartSLOduration=131.463699197 podStartE2EDuration="2m12.762024257s" totalImagesPullingTime="1.29832506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:45.76148086 +0000 UTC m=+133.106589479" watchObservedRunningTime="2026-09-15 11:40:45.762024257 +0000 UTC m=+133.107132863" Sep 15 11:41:00.795367 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:41:00.795277 2579 generic.go:356] "Generic (PLEG): container finished" podID="d657879d-7cf6-4d6c-82b8-71fb715059e3" containerID="b293cc57766fcd215e4cfc8d50cb982a060069c947746f2dcdad645c9dfaf0b3" exitCode=0 Sep 15 11:41:00.795367 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:41:00.795351 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mxrhs" event={"ID":"d657879d-7cf6-4d6c-82b8-71fb715059e3","Type":"ContainerDied","Data":"b293cc57766fcd215e4cfc8d50cb982a060069c947746f2dcdad645c9dfaf0b3"} Sep 15 11:41:00.795782 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:41:00.795715 2579 scope.go:117] "RemoveContainer" containerID="b293cc57766fcd215e4cfc8d50cb982a060069c947746f2dcdad645c9dfaf0b3" Sep 15 11:41:01.800888 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:41:01.800851 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-mxrhs" event={"ID":"d657879d-7cf6-4d6c-82b8-71fb715059e3","Type":"ContainerStarted","Data":"7d96f28f4af0f72d7266c5e1471cd3fea5c5cf1cc16f9dff722a571ade89d3b8"} Sep 15 11:43:33.082815 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:43:33.082787 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:43:33.089585 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:43:33.089561 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:43:33.092493 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:43:33.092470 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 15 11:44:24.338682 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.338606 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5dcj8"] Sep 15 11:44:24.342416 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.342397 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:24.344632 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.344613 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 15 11:44:24.344725 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.344650 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 15 11:44:24.353074 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.353053 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5dcj8"] Sep 15 11:44:24.403615 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.403579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm864\" (UniqueName: \"kubernetes.io/projected/681cd17e-96cf-40af-892e-db6a1ae5d97f-kube-api-access-jm864\") pod \"cert-manager-webhook-6486ffd997-5dcj8\" (UID: \"681cd17e-96cf-40af-892e-db6a1ae5d97f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:24.403760 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.403617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/681cd17e-96cf-40af-892e-db6a1ae5d97f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5dcj8\" (UID: \"681cd17e-96cf-40af-892e-db6a1ae5d97f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:24.504011 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.503974 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm864\" (UniqueName: \"kubernetes.io/projected/681cd17e-96cf-40af-892e-db6a1ae5d97f-kube-api-access-jm864\") pod \"cert-manager-webhook-6486ffd997-5dcj8\" (UID: \"681cd17e-96cf-40af-892e-db6a1ae5d97f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:24.504011 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.504014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/681cd17e-96cf-40af-892e-db6a1ae5d97f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5dcj8\" (UID: \"681cd17e-96cf-40af-892e-db6a1ae5d97f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:24.516872 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.516847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/681cd17e-96cf-40af-892e-db6a1ae5d97f-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5dcj8\" (UID: \"681cd17e-96cf-40af-892e-db6a1ae5d97f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:24.516981 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.516893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm864\" (UniqueName: \"kubernetes.io/projected/681cd17e-96cf-40af-892e-db6a1ae5d97f-kube-api-access-jm864\") pod \"cert-manager-webhook-6486ffd997-5dcj8\" (UID: \"681cd17e-96cf-40af-892e-db6a1ae5d97f\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:24.650927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.650830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:24.929621 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.929590 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5dcj8"] Sep 15 11:44:24.932557 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:44:24.932530 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod681cd17e_96cf_40af_892e_db6a1ae5d97f.slice/crio-db527a97bcd9cd08233b2ab713cc9c21ba594d23b5d642f33a1fff9adb8f1617 WatchSource:0}: Error finding container db527a97bcd9cd08233b2ab713cc9c21ba594d23b5d642f33a1fff9adb8f1617: Status 404 returned error can't find the container with id db527a97bcd9cd08233b2ab713cc9c21ba594d23b5d642f33a1fff9adb8f1617 Sep 15 11:44:24.934381 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:24.934365 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 11:44:25.490088 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:25.490053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" event={"ID":"681cd17e-96cf-40af-892e-db6a1ae5d97f","Type":"ContainerStarted","Data":"db527a97bcd9cd08233b2ab713cc9c21ba594d23b5d642f33a1fff9adb8f1617"} Sep 15 11:44:28.503097 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:28.503059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" event={"ID":"681cd17e-96cf-40af-892e-db6a1ae5d97f","Type":"ContainerStarted","Data":"2220284ccf75c40530d07c10a84781e5287d09feb82967d16fdda8c2faaddcf1"} Sep 15 11:44:28.503579 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:28.503173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:44:28.520130 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:28.520054 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" podStartSLOduration=1.312065755 podStartE2EDuration="4.520038928s" totalImagesPullingTime="3.207973173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:28.518651998 +0000 UTC m=+355.863760603" watchObservedRunningTime="2026-09-15 11:44:28.520038928 +0000 UTC m=+355.865147511" Sep 15 11:44:34.508648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:44:34.508614 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5dcj8" Sep 15 11:45:03.043948 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.043913 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq"] Sep 15 11:45:03.046704 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.046685 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.048954 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.048932 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 15 11:45:03.049132 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.049112 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 15 11:45:03.049228 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.049156 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 15 11:45:03.049228 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.049172 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 15 11:45:03.049343 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.049324 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bx677\"" Sep 15 11:45:03.060254 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.060231 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq"] Sep 15 11:45:03.083808 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.083767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5204c2ed-93f0-40c7-a024-855959b4f0a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.083967 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.083870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5204c2ed-93f0-40c7-a024-855959b4f0a0-tmp\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.083967 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.083920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5204c2ed-93f0-40c7-a024-855959b4f0a0-manifests\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.083967 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.083948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5204c2ed-93f0-40c7-a024-855959b4f0a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.084100 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.083974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmlg\" (UniqueName: \"kubernetes.io/projected/5204c2ed-93f0-40c7-a024-855959b4f0a0-kube-api-access-fzmlg\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.184434 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.184397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5204c2ed-93f0-40c7-a024-855959b4f0a0-tmp\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.184434 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.184442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5204c2ed-93f0-40c7-a024-855959b4f0a0-manifests\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.184672 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.184461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5204c2ed-93f0-40c7-a024-855959b4f0a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.184672 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.184565 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmlg\" (UniqueName: \"kubernetes.io/projected/5204c2ed-93f0-40c7-a024-855959b4f0a0-kube-api-access-fzmlg\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.184672 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.184614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5204c2ed-93f0-40c7-a024-855959b4f0a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.184832 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.184812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5204c2ed-93f0-40c7-a024-855959b4f0a0-manifests\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.187038 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.187008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5204c2ed-93f0-40c7-a024-855959b4f0a0-tmp\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.187245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.187230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5204c2ed-93f0-40c7-a024-855959b4f0a0-webhook-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.187400 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.187382 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5204c2ed-93f0-40c7-a024-855959b4f0a0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.193485 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.193463 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmlg\" (UniqueName: \"kubernetes.io/projected/5204c2ed-93f0-40c7-a024-855959b4f0a0-kube-api-access-fzmlg\") pod \"opendatahub-operator-controller-manager-66bc7bc78c-lpkmq\" (UID: \"5204c2ed-93f0-40c7-a024-855959b4f0a0\") " pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.356517 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.356424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:03.631629 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:03.631505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq"] Sep 15 11:45:03.636107 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:45:03.636082 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5204c2ed_93f0_40c7_a024_855959b4f0a0.slice/crio-301e4e45cc7e3eedf42c159c2b1313fa89c23718c33aaf65a476f19574d9f1ef WatchSource:0}: Error finding container 301e4e45cc7e3eedf42c159c2b1313fa89c23718c33aaf65a476f19574d9f1ef: Status 404 returned error can't find the container with id 301e4e45cc7e3eedf42c159c2b1313fa89c23718c33aaf65a476f19574d9f1ef Sep 15 11:45:04.626673 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:04.626598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" event={"ID":"5204c2ed-93f0-40c7-a024-855959b4f0a0","Type":"ContainerStarted","Data":"301e4e45cc7e3eedf42c159c2b1313fa89c23718c33aaf65a476f19574d9f1ef"} Sep 15 11:45:07.645963 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:07.645910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" event={"ID":"5204c2ed-93f0-40c7-a024-855959b4f0a0","Type":"ContainerStarted","Data":"c08f9d245bc03b76cbc7a764cd23b14e182ced8c89675e581cb2f3fae9284fa6"} Sep 15 11:45:08.652779 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:08.652742 2579 generic.go:356] "Generic (PLEG): container finished" podID="5204c2ed-93f0-40c7-a024-855959b4f0a0" containerID="c08f9d245bc03b76cbc7a764cd23b14e182ced8c89675e581cb2f3fae9284fa6" exitCode=0 Sep 15 11:45:08.653146 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:08.652829 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" event={"ID":"5204c2ed-93f0-40c7-a024-855959b4f0a0","Type":"ContainerDied","Data":"c08f9d245bc03b76cbc7a764cd23b14e182ced8c89675e581cb2f3fae9284fa6"} Sep 15 11:45:09.657912 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:09.657880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" event={"ID":"5204c2ed-93f0-40c7-a024-855959b4f0a0","Type":"ContainerStarted","Data":"e839eb5070e144a21ad00d61ef02ad8ee20672c6811233368e6426236a5a4b9d"} Sep 15 11:45:09.658299 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:09.658019 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:09.683994 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:09.683939 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" podStartSLOduration=2.3389226499999998 podStartE2EDuration="6.683919391s" totalImagesPullingTime="3.307707101s" totalInitContainerRuntime="1.03728964s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:09.682473864 +0000 UTC m=+397.027582469" watchObservedRunningTime="2026-09-15 11:45:09.683919391 +0000 UTC m=+397.029027992" Sep 15 11:45:20.663703 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:20.663677 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-66bc7bc78c-lpkmq" Sep 15 11:45:22.711098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.711064 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9"] Sep 15 11:45:22.722715 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.722681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.725166 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.724989 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 15 11:45:22.725323 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.725177 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 15 11:45:22.725380 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.725332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-hm6dh\"" Sep 15 11:45:22.725380 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.725333 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9"] Sep 15 11:45:22.725499 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.725486 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 15 11:45:22.725854 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.725837 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 15 11:45:22.827913 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.827879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdd1b29-a854-4729-9b69-b1819af7d102-tls-certs\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.828098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.827924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efdd1b29-a854-4729-9b69-b1819af7d102-tmp\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.828098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.828062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p7rf\" (UniqueName: \"kubernetes.io/projected/efdd1b29-a854-4729-9b69-b1819af7d102-kube-api-access-7p7rf\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.929257 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.929218 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p7rf\" (UniqueName: \"kubernetes.io/projected/efdd1b29-a854-4729-9b69-b1819af7d102-kube-api-access-7p7rf\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.929257 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.929265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdd1b29-a854-4729-9b69-b1819af7d102-tls-certs\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.929451 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.929291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efdd1b29-a854-4729-9b69-b1819af7d102-tmp\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.931651 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.931623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efdd1b29-a854-4729-9b69-b1819af7d102-tmp\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.931852 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.931830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efdd1b29-a854-4729-9b69-b1819af7d102-tls-certs\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:22.939109 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:22.939088 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p7rf\" (UniqueName: \"kubernetes.io/projected/efdd1b29-a854-4729-9b69-b1819af7d102-kube-api-access-7p7rf\") pod \"kube-auth-proxy-df8dfc748-4sgj9\" (UID: \"efdd1b29-a854-4729-9b69-b1819af7d102\") " pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:23.033959 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.033883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" Sep 15 11:45:23.313153 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.313072 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9"] Sep 15 11:45:23.315827 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:45:23.315791 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdd1b29_a854_4729_9b69_b1819af7d102.slice/crio-8d7df9a6534b8a3f9c30ef2744e47453b7d2e5bb401fc06b8b7ff7473bab50c7 WatchSource:0}: Error finding container 8d7df9a6534b8a3f9c30ef2744e47453b7d2e5bb401fc06b8b7ff7473bab50c7: Status 404 returned error can't find the container with id 8d7df9a6534b8a3f9c30ef2744e47453b7d2e5bb401fc06b8b7ff7473bab50c7 Sep 15 11:45:23.706906 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.706873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" event={"ID":"efdd1b29-a854-4729-9b69-b1819af7d102","Type":"ContainerStarted","Data":"8d7df9a6534b8a3f9c30ef2744e47453b7d2e5bb401fc06b8b7ff7473bab50c7"} Sep 15 11:45:23.763744 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.763709 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr"] Sep 15 11:45:23.766266 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.766249 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.768710 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.768685 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 15 11:45:23.768829 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.768738 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 15 11:45:23.768829 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.768744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sd7pv\"" Sep 15 11:45:23.768829 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.768770 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 15 11:45:23.768829 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.768788 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 15 11:45:23.778037 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.778015 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr"] Sep 15 11:45:23.835705 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.835668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.835705 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.835706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.835927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.835742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.835927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.835785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.835927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.835806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmbnf\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-kube-api-access-nmbnf\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.835927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.835847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.835927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.835879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71f3849-4527-4888-91b2-058e8a3e7c8a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.936500 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.936359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.936500 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.936485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.937135 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.936536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.937135 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.936571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.937135 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.936594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmbnf\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-kube-api-access-nmbnf\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.937135 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.936624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.937135 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.936705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71f3849-4527-4888-91b2-058e8a3e7c8a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.937409 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.937365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.942557 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.942519 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.942557 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.942527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.942959 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.942919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71f3849-4527-4888-91b2-058e8a3e7c8a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.944401 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.944353 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.945426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.945387 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:23.949314 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:23.949184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmbnf\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-kube-api-access-nmbnf\") pod \"istiod-openshift-gateway-7b7995d5cd-xvmpr\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:24.075423 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:24.075342 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:24.366473 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:24.366444 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr"] Sep 15 11:45:24.368944 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:45:24.368867 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71f3849_4527_4888_91b2_058e8a3e7c8a.slice/crio-ae0f2591de262f195c0a7f2cba788fa63b24e873dcbb720b9ff2edcb85138b8f WatchSource:0}: Error finding container ae0f2591de262f195c0a7f2cba788fa63b24e873dcbb720b9ff2edcb85138b8f: Status 404 returned error can't find the container with id ae0f2591de262f195c0a7f2cba788fa63b24e873dcbb720b9ff2edcb85138b8f Sep 15 11:45:24.713262 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:24.713227 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" event={"ID":"b71f3849-4527-4888-91b2-058e8a3e7c8a","Type":"ContainerStarted","Data":"ae0f2591de262f195c0a7f2cba788fa63b24e873dcbb720b9ff2edcb85138b8f"} Sep 15 11:45:26.722578 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:26.722536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" event={"ID":"efdd1b29-a854-4729-9b69-b1819af7d102","Type":"ContainerStarted","Data":"329656f7de81e2550e7c50163a946193e9e440d6a920333067f5adadbf584d9a"} Sep 15 11:45:26.742472 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:26.742416 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-df8dfc748-4sgj9" podStartSLOduration=2.142871547 podStartE2EDuration="4.742395878s" totalImagesPullingTime="2.599524331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:26.741094822 +0000 UTC m=+414.086203428" watchObservedRunningTime="2026-09-15 11:45:26.742395878 +0000 UTC m=+414.087504485" Sep 15 11:45:27.639835 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:27.639789 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 15 11:45:27.639938 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:27.639890 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 15 11:45:27.728491 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:27.728460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" event={"ID":"b71f3849-4527-4888-91b2-058e8a3e7c8a","Type":"ContainerStarted","Data":"70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad"} Sep 15 11:45:27.728847 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:27.728560 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:27.751763 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:27.751716 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" podStartSLOduration=1.483840685 podStartE2EDuration="4.751698188s" totalImagesPullingTime="3.267857503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:27.750023427 +0000 UTC m=+415.095132033" watchObservedRunningTime="2026-09-15 11:45:27.751698188 +0000 UTC m=+415.096806793" Sep 15 11:45:28.733965 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:28.733925 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-xvmpr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 15 11:45:28.734359 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:28.734000 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" podUID="b71f3849-4527-4888-91b2-058e8a3e7c8a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 15 11:45:31.733423 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:31.733395 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:43.420101 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.420070 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lknpt"] Sep 15 11:45:43.424538 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.424517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:43.426963 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.426947 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 15 11:45:43.427074 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.426987 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lt78t\"" Sep 15 11:45:43.436794 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.436757 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lknpt"] Sep 15 11:45:43.438096 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.438077 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf"] Sep 15 11:45:43.441560 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.441539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:43.444037 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.444010 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-8w87j\"" Sep 15 11:45:43.444267 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.444242 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 15 11:45:43.452742 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.452721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf"] Sep 15 11:45:43.498653 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.498620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkv6\" (UniqueName: \"kubernetes.io/projected/83dbb156-d8db-496a-8b6d-273d8b71a7d6-kube-api-access-jfkv6\") pod \"llmisvc-controller-manager-777c6bd7d5-n2zwf\" (UID: \"83dbb156-d8db-496a-8b6d-273d8b71a7d6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:43.498807 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.498660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/621c0998-47bc-4da1-8be5-b20fdebdef73-cert\") pod \"kserve-controller-manager-d6f4b65b8-lknpt\" (UID: \"621c0998-47bc-4da1-8be5-b20fdebdef73\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:43.498807 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.498716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz6s\" (UniqueName: \"kubernetes.io/projected/621c0998-47bc-4da1-8be5-b20fdebdef73-kube-api-access-zvz6s\") pod \"kserve-controller-manager-d6f4b65b8-lknpt\" (UID: \"621c0998-47bc-4da1-8be5-b20fdebdef73\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:43.498807 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.498744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83dbb156-d8db-496a-8b6d-273d8b71a7d6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n2zwf\" (UID: \"83dbb156-d8db-496a-8b6d-273d8b71a7d6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:43.599250 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.599215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83dbb156-d8db-496a-8b6d-273d8b71a7d6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n2zwf\" (UID: \"83dbb156-d8db-496a-8b6d-273d8b71a7d6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:43.599446 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.599279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkv6\" (UniqueName: \"kubernetes.io/projected/83dbb156-d8db-496a-8b6d-273d8b71a7d6-kube-api-access-jfkv6\") pod \"llmisvc-controller-manager-777c6bd7d5-n2zwf\" (UID: \"83dbb156-d8db-496a-8b6d-273d8b71a7d6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:43.599446 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.599294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/621c0998-47bc-4da1-8be5-b20fdebdef73-cert\") pod \"kserve-controller-manager-d6f4b65b8-lknpt\" (UID: \"621c0998-47bc-4da1-8be5-b20fdebdef73\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:43.599446 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.599336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz6s\" (UniqueName: \"kubernetes.io/projected/621c0998-47bc-4da1-8be5-b20fdebdef73-kube-api-access-zvz6s\") pod \"kserve-controller-manager-d6f4b65b8-lknpt\" (UID: \"621c0998-47bc-4da1-8be5-b20fdebdef73\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:43.599446 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:45:43.599383 2579 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 15 11:45:43.599446 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:45:43.599448 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83dbb156-d8db-496a-8b6d-273d8b71a7d6-cert podName:83dbb156-d8db-496a-8b6d-273d8b71a7d6 nodeName:}" failed. No retries permitted until 2026-09-15 11:45:44.099433326 +0000 UTC m=+431.444541912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/83dbb156-d8db-496a-8b6d-273d8b71a7d6-cert") pod "llmisvc-controller-manager-777c6bd7d5-n2zwf" (UID: "83dbb156-d8db-496a-8b6d-273d8b71a7d6") : secret "llmisvc-webhook-server-cert" not found Sep 15 11:45:43.599633 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:45:43.599449 2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 15 11:45:43.599633 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:45:43.599506 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/621c0998-47bc-4da1-8be5-b20fdebdef73-cert podName:621c0998-47bc-4da1-8be5-b20fdebdef73 nodeName:}" failed. No retries permitted until 2026-09-15 11:45:44.099492383 +0000 UTC m=+431.444600966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/621c0998-47bc-4da1-8be5-b20fdebdef73-cert") pod "kserve-controller-manager-d6f4b65b8-lknpt" (UID: "621c0998-47bc-4da1-8be5-b20fdebdef73") : secret "kserve-webhook-server-cert" not found Sep 15 11:45:43.612127 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.612097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz6s\" (UniqueName: \"kubernetes.io/projected/621c0998-47bc-4da1-8be5-b20fdebdef73-kube-api-access-zvz6s\") pod \"kserve-controller-manager-d6f4b65b8-lknpt\" (UID: \"621c0998-47bc-4da1-8be5-b20fdebdef73\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:43.613051 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:43.613033 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkv6\" (UniqueName: \"kubernetes.io/projected/83dbb156-d8db-496a-8b6d-273d8b71a7d6-kube-api-access-jfkv6\") pod \"llmisvc-controller-manager-777c6bd7d5-n2zwf\" (UID: \"83dbb156-d8db-496a-8b6d-273d8b71a7d6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:44.103723 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.103693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/621c0998-47bc-4da1-8be5-b20fdebdef73-cert\") pod \"kserve-controller-manager-d6f4b65b8-lknpt\" (UID: \"621c0998-47bc-4da1-8be5-b20fdebdef73\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:44.103887 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.103740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83dbb156-d8db-496a-8b6d-273d8b71a7d6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n2zwf\" (UID: \"83dbb156-d8db-496a-8b6d-273d8b71a7d6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:44.106280 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.106250 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/621c0998-47bc-4da1-8be5-b20fdebdef73-cert\") pod \"kserve-controller-manager-d6f4b65b8-lknpt\" (UID: \"621c0998-47bc-4da1-8be5-b20fdebdef73\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:44.106383 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.106284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83dbb156-d8db-496a-8b6d-273d8b71a7d6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-n2zwf\" (UID: \"83dbb156-d8db-496a-8b6d-273d8b71a7d6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:44.334090 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.334049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:44.351321 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.351297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:44.608832 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.608708 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lknpt"] Sep 15 11:45:44.612536 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:45:44.612502 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621c0998_47bc_4da1_8be5_b20fdebdef73.slice/crio-8cf4b0b1a6f4c6c67f21e6522ba1fe4e7055105468bc47d6af4888e7c9cc1b03 WatchSource:0}: Error finding container 8cf4b0b1a6f4c6c67f21e6522ba1fe4e7055105468bc47d6af4888e7c9cc1b03: Status 404 returned error can't find the container with id 8cf4b0b1a6f4c6c67f21e6522ba1fe4e7055105468bc47d6af4888e7c9cc1b03 Sep 15 11:45:44.628447 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.628413 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf"] Sep 15 11:45:44.630960 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:45:44.630937 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod83dbb156_d8db_496a_8b6d_273d8b71a7d6.slice/crio-d04189db3330d5b3dcbab818b1c107f65478295e5d8ce4915d9fa2668c8f731c WatchSource:0}: Error finding container d04189db3330d5b3dcbab818b1c107f65478295e5d8ce4915d9fa2668c8f731c: Status 404 returned error can't find the container with id d04189db3330d5b3dcbab818b1c107f65478295e5d8ce4915d9fa2668c8f731c Sep 15 11:45:44.790662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.790619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" event={"ID":"621c0998-47bc-4da1-8be5-b20fdebdef73","Type":"ContainerStarted","Data":"8cf4b0b1a6f4c6c67f21e6522ba1fe4e7055105468bc47d6af4888e7c9cc1b03"} Sep 15 11:45:44.791807 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:44.791782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" event={"ID":"83dbb156-d8db-496a-8b6d-273d8b71a7d6","Type":"ContainerStarted","Data":"d04189db3330d5b3dcbab818b1c107f65478295e5d8ce4915d9fa2668c8f731c"} Sep 15 11:45:45.900482 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:45.900082 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr"] Sep 15 11:45:45.900482 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:45.900347 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" podUID="b71f3849-4527-4888-91b2-058e8a3e7c8a" containerName="discovery" containerID="cri-o://70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad" gracePeriod=30 Sep 15 11:45:46.691817 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.691782 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:46.803288 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.803257 2579 generic.go:356] "Generic (PLEG): container finished" podID="b71f3849-4527-4888-91b2-058e8a3e7c8a" containerID="70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad" exitCode=0 Sep 15 11:45:46.803428 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.803316 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" Sep 15 11:45:46.803428 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.803340 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" event={"ID":"b71f3849-4527-4888-91b2-058e8a3e7c8a","Type":"ContainerDied","Data":"70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad"} Sep 15 11:45:46.803428 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.803418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr" event={"ID":"b71f3849-4527-4888-91b2-058e8a3e7c8a","Type":"ContainerDied","Data":"ae0f2591de262f195c0a7f2cba788fa63b24e873dcbb720b9ff2edcb85138b8f"} Sep 15 11:45:46.803585 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.803447 2579 scope.go:117] "RemoveContainer" containerID="70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad" Sep 15 11:45:46.813150 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.813127 2579 scope.go:117] "RemoveContainer" containerID="70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad" Sep 15 11:45:46.813500 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:45:46.813471 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad\": container with ID starting with 70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad not found: ID does not exist" containerID="70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad" Sep 15 11:45:46.813609 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.813508 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad"} err="failed to get container status \"70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad\": rpc error: code = NotFound desc = could not find container \"70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad\": container with ID starting with 70fad846310bad53eb1d6c31971b94159dd86fb00853101bf2ed4a054313ccad not found: ID does not exist" Sep 15 11:45:46.826246 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.826195 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-token\") pod \"b71f3849-4527-4888-91b2-058e8a3e7c8a\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " Sep 15 11:45:46.826401 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.826298 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-kubeconfig\") pod \"b71f3849-4527-4888-91b2-058e8a3e7c8a\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " Sep 15 11:45:46.826401 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.826323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-dns-cert\") pod \"b71f3849-4527-4888-91b2-058e8a3e7c8a\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " Sep 15 11:45:46.826401 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.826350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71f3849-4527-4888-91b2-058e8a3e7c8a-local-certs\") pod \"b71f3849-4527-4888-91b2-058e8a3e7c8a\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " Sep 15 11:45:46.826401 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.826382 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmbnf\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-kube-api-access-nmbnf\") pod \"b71f3849-4527-4888-91b2-058e8a3e7c8a\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " Sep 15 11:45:46.826685 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.826421 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-ca-configmap\") pod \"b71f3849-4527-4888-91b2-058e8a3e7c8a\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " Sep 15 11:45:46.826685 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.826448 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-cacerts\") pod \"b71f3849-4527-4888-91b2-058e8a3e7c8a\" (UID: \"b71f3849-4527-4888-91b2-058e8a3e7c8a\") " Sep 15 11:45:46.827120 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.827014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b71f3849-4527-4888-91b2-058e8a3e7c8a" (UID: "b71f3849-4527-4888-91b2-058e8a3e7c8a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:45:46.829411 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.829367 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-token" (OuterVolumeSpecName: "istio-token") pod "b71f3849-4527-4888-91b2-058e8a3e7c8a" (UID: "b71f3849-4527-4888-91b2-058e8a3e7c8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:45:46.829879 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.829836 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-cacerts" (OuterVolumeSpecName: "cacerts") pod "b71f3849-4527-4888-91b2-058e8a3e7c8a" (UID: "b71f3849-4527-4888-91b2-058e8a3e7c8a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:45:46.830081 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.830055 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b71f3849-4527-4888-91b2-058e8a3e7c8a" (UID: "b71f3849-4527-4888-91b2-058e8a3e7c8a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:45:46.830081 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.830055 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71f3849-4527-4888-91b2-058e8a3e7c8a-local-certs" (OuterVolumeSpecName: "local-certs") pod "b71f3849-4527-4888-91b2-058e8a3e7c8a" (UID: "b71f3849-4527-4888-91b2-058e8a3e7c8a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:45:46.830289 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.830103 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b71f3849-4527-4888-91b2-058e8a3e7c8a" (UID: "b71f3849-4527-4888-91b2-058e8a3e7c8a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:45:46.830369 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.830348 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-kube-api-access-nmbnf" (OuterVolumeSpecName: "kube-api-access-nmbnf") pod "b71f3849-4527-4888-91b2-058e8a3e7c8a" (UID: "b71f3849-4527-4888-91b2-058e8a3e7c8a"). InnerVolumeSpecName "kube-api-access-nmbnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:45:46.927919 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.927893 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-kubeconfig\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:45:46.928378 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.927923 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-dns-cert\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:45:46.928378 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.927933 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71f3849-4527-4888-91b2-058e8a3e7c8a-local-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:45:46.928378 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.927943 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmbnf\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-kube-api-access-nmbnf\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:45:46.928378 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.927952 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-csr-ca-configmap\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:45:46.928378 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.927960 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b71f3849-4527-4888-91b2-058e8a3e7c8a-cacerts\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:45:46.928378 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:46.927967 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71f3849-4527-4888-91b2-058e8a3e7c8a-istio-token\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:45:47.140040 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.140000 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr"] Sep 15 11:45:47.147658 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.147629 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-xvmpr"] Sep 15 11:45:47.187822 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.187742 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71f3849-4527-4888-91b2-058e8a3e7c8a" path="/var/lib/kubelet/pods/b71f3849-4527-4888-91b2-058e8a3e7c8a/volumes" Sep 15 11:45:47.808468 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.808364 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" event={"ID":"621c0998-47bc-4da1-8be5-b20fdebdef73","Type":"ContainerStarted","Data":"73a8e7ff1899849bd9e519a0a6b2113d55b52de695ab76dcafe313a849e6835e"} Sep 15 11:45:47.808633 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.808475 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:45:47.810636 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.810603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" event={"ID":"83dbb156-d8db-496a-8b6d-273d8b71a7d6","Type":"ContainerStarted","Data":"5b28c84a0a92ee0bcbf16193ae578eff540c7344a0ff76daff6ec591cc458c6a"} Sep 15 11:45:47.810752 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.810658 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:45:47.826294 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.826246 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" podStartSLOduration=2.684612295 podStartE2EDuration="4.826227558s" totalImagesPullingTime="2.141615263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:47.825329889 +0000 UTC m=+435.170438495" watchObservedRunningTime="2026-09-15 11:45:47.826227558 +0000 UTC m=+435.171336159" Sep 15 11:45:47.845924 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:45:47.845880 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" podStartSLOduration=2.022161946 podStartE2EDuration="4.845864784s" totalImagesPullingTime="2.823702838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:47.844539316 +0000 UTC m=+435.189647933" watchObservedRunningTime="2026-09-15 11:45:47.845864784 +0000 UTC m=+435.190973390" Sep 15 11:46:18.817024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:18.816993 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-n2zwf" Sep 15 11:46:18.819831 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:18.819816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lknpt" Sep 15 11:46:58.544990 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.544958 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-znvd5"] Sep 15 11:46:58.545444 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.545264 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71f3849-4527-4888-91b2-058e8a3e7c8a" containerName="discovery" Sep 15 11:46:58.545444 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.545279 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71f3849-4527-4888-91b2-058e8a3e7c8a" containerName="discovery" Sep 15 11:46:58.545444 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.545341 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71f3849-4527-4888-91b2-058e8a3e7c8a" containerName="discovery" Sep 15 11:46:58.547926 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.547912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" Sep 15 11:46:58.550182 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.550163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xx2h9\"" Sep 15 11:46:58.550329 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.550181 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 15 11:46:58.550329 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.550214 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 15 11:46:58.559606 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.559584 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-znvd5"] Sep 15 11:46:58.644167 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.644125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5brm7\" (UniqueName: \"kubernetes.io/projected/93df8817-6a9f-4663-bb6b-2fc8d6981b97-kube-api-access-5brm7\") pod \"authorino-operator-76b478bd78-znvd5\" (UID: \"93df8817-6a9f-4663-bb6b-2fc8d6981b97\") " pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" Sep 15 11:46:58.745301 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.745262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5brm7\" (UniqueName: \"kubernetes.io/projected/93df8817-6a9f-4663-bb6b-2fc8d6981b97-kube-api-access-5brm7\") pod \"authorino-operator-76b478bd78-znvd5\" (UID: \"93df8817-6a9f-4663-bb6b-2fc8d6981b97\") " pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" Sep 15 11:46:58.757829 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.757799 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5brm7\" (UniqueName: \"kubernetes.io/projected/93df8817-6a9f-4663-bb6b-2fc8d6981b97-kube-api-access-5brm7\") pod \"authorino-operator-76b478bd78-znvd5\" (UID: \"93df8817-6a9f-4663-bb6b-2fc8d6981b97\") " pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" Sep 15 11:46:58.856861 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:58.856771 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" Sep 15 11:46:59.130547 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:46:59.130474 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-znvd5"] Sep 15 11:46:59.134118 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:46:59.134091 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93df8817_6a9f_4663_bb6b_2fc8d6981b97.slice/crio-a19ff72ffb008ecf65e2767741f6ed7596f7e364a17e32c4ab435d929f5e7007 WatchSource:0}: Error finding container a19ff72ffb008ecf65e2767741f6ed7596f7e364a17e32c4ab435d929f5e7007: Status 404 returned error can't find the container with id a19ff72ffb008ecf65e2767741f6ed7596f7e364a17e32c4ab435d929f5e7007 Sep 15 11:47:00.081929 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:00.081899 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" event={"ID":"93df8817-6a9f-4663-bb6b-2fc8d6981b97","Type":"ContainerStarted","Data":"a19ff72ffb008ecf65e2767741f6ed7596f7e364a17e32c4ab435d929f5e7007"} Sep 15 11:47:00.856462 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:00.856427 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn"] Sep 15 11:47:00.859889 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:00.859863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:00.862276 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:00.862255 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mgz55\"" Sep 15 11:47:00.871795 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:00.871773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn"] Sep 15 11:47:00.961477 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:00.961431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b929b\" (UniqueName: \"kubernetes.io/projected/67b8f7bf-551b-4e8a-8ca1-574d8ff17436-kube-api-access-b929b\") pod \"limitador-operator-controller-manager-74495c69f8-rr4hn\" (UID: \"67b8f7bf-551b-4e8a-8ca1-574d8ff17436\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:01.062548 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:01.062508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b929b\" (UniqueName: \"kubernetes.io/projected/67b8f7bf-551b-4e8a-8ca1-574d8ff17436-kube-api-access-b929b\") pod \"limitador-operator-controller-manager-74495c69f8-rr4hn\" (UID: \"67b8f7bf-551b-4e8a-8ca1-574d8ff17436\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:01.075730 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:01.075671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b929b\" (UniqueName: \"kubernetes.io/projected/67b8f7bf-551b-4e8a-8ca1-574d8ff17436-kube-api-access-b929b\") pod \"limitador-operator-controller-manager-74495c69f8-rr4hn\" (UID: \"67b8f7bf-551b-4e8a-8ca1-574d8ff17436\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:01.172330 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:01.172248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:02.091997 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:02.091962 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" event={"ID":"93df8817-6a9f-4663-bb6b-2fc8d6981b97","Type":"ContainerStarted","Data":"7df87b0a94fc1e8837d9dc78c8314d6106f7654a8d39390673822b86b891094e"} Sep 15 11:47:02.092186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:02.092067 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" Sep 15 11:47:02.110903 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:02.110814 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" podStartSLOduration=1.320852006 podStartE2EDuration="4.110796479s" totalImagesPullingTime="2.789944473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:02.109313063 +0000 UTC m=+509.454421672" watchObservedRunningTime="2026-09-15 11:47:02.110796479 +0000 UTC m=+509.455905083" Sep 15 11:47:02.118242 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:02.118196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn"] Sep 15 11:47:02.123125 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:02.123095 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b8f7bf_551b_4e8a_8ca1_574d8ff17436.slice/crio-8db117fbc041433e21b43ddc780a8cbee9c09a63f943b6a03d8b60835a2c32a6 WatchSource:0}: Error finding container 8db117fbc041433e21b43ddc780a8cbee9c09a63f943b6a03d8b60835a2c32a6: Status 404 returned error can't find the container with id 8db117fbc041433e21b43ddc780a8cbee9c09a63f943b6a03d8b60835a2c32a6 Sep 15 11:47:03.060045 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.060006 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv"] Sep 15 11:47:03.062542 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.062519 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" Sep 15 11:47:03.064774 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.064747 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 15 11:47:03.064882 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.064786 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-48mwk\"" Sep 15 11:47:03.074989 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.074968 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv"] Sep 15 11:47:03.097354 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.097323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" event={"ID":"67b8f7bf-551b-4e8a-8ca1-574d8ff17436","Type":"ContainerStarted","Data":"8db117fbc041433e21b43ddc780a8cbee9c09a63f943b6a03d8b60835a2c32a6"} Sep 15 11:47:03.180756 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.180712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdj5z\" (UniqueName: \"kubernetes.io/projected/d069f28d-a8d6-4393-af3c-748c149d934f-kube-api-access-tdj5z\") pod \"dns-operator-controller-manager-7b84cd7c9c-fgnnv\" (UID: \"d069f28d-a8d6-4393-af3c-748c149d934f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" Sep 15 11:47:03.281348 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.281322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdj5z\" (UniqueName: \"kubernetes.io/projected/d069f28d-a8d6-4393-af3c-748c149d934f-kube-api-access-tdj5z\") pod \"dns-operator-controller-manager-7b84cd7c9c-fgnnv\" (UID: \"d069f28d-a8d6-4393-af3c-748c149d934f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" Sep 15 11:47:03.290641 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.290610 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdj5z\" (UniqueName: \"kubernetes.io/projected/d069f28d-a8d6-4393-af3c-748c149d934f-kube-api-access-tdj5z\") pod \"dns-operator-controller-manager-7b84cd7c9c-fgnnv\" (UID: \"d069f28d-a8d6-4393-af3c-748c149d934f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" Sep 15 11:47:03.372342 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.372256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" Sep 15 11:47:03.647619 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:03.647545 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv"] Sep 15 11:47:03.650763 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:03.650732 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd069f28d_a8d6_4393_af3c_748c149d934f.slice/crio-aa2c4aff5351a7a2dd59589314fbc302845aab11724761b701a458cb9eedbd46 WatchSource:0}: Error finding container aa2c4aff5351a7a2dd59589314fbc302845aab11724761b701a458cb9eedbd46: Status 404 returned error can't find the container with id aa2c4aff5351a7a2dd59589314fbc302845aab11724761b701a458cb9eedbd46 Sep 15 11:47:04.103279 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:04.103245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" event={"ID":"d069f28d-a8d6-4393-af3c-748c149d934f","Type":"ContainerStarted","Data":"aa2c4aff5351a7a2dd59589314fbc302845aab11724761b701a458cb9eedbd46"} Sep 15 11:47:05.109489 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:05.109452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" event={"ID":"67b8f7bf-551b-4e8a-8ca1-574d8ff17436","Type":"ContainerStarted","Data":"ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85"} Sep 15 11:47:05.109949 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:05.109608 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:05.129346 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:05.129301 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" podStartSLOduration=2.993018885 podStartE2EDuration="5.129286506s" totalImagesPullingTime="2.136267621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:05.128092966 +0000 UTC m=+512.473201572" watchObservedRunningTime="2026-09-15 11:47:05.129286506 +0000 UTC m=+512.474395110" Sep 15 11:47:06.116103 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:06.116059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" event={"ID":"d069f28d-a8d6-4393-af3c-748c149d934f","Type":"ContainerStarted","Data":"1c423eb06e002bf857cab7e6cf9f4c47a04c24e22ee2039597933ff3e2b6f9e0"} Sep 15 11:47:06.136046 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:06.135991 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" podStartSLOduration=1.440019533 podStartE2EDuration="3.135973973s" totalImagesPullingTime="1.69595444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:06.133810328 +0000 UTC m=+513.478918933" watchObservedRunningTime="2026-09-15 11:47:06.135973973 +0000 UTC m=+513.481082577" Sep 15 11:47:07.121725 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.121683 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" Sep 15 11:47:07.513785 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.513740 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w"] Sep 15 11:47:07.517037 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.517017 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:07.519614 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.519593 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xwdp5\"" Sep 15 11:47:07.530874 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.530853 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w"] Sep 15 11:47:07.614838 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.614798 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784b3e4a-c33f-4330-b0c9-f77078833fd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" (UID: \"784b3e4a-c33f-4330-b0c9-f77078833fd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:07.615023 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.614916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42q7n\" (UniqueName: \"kubernetes.io/projected/784b3e4a-c33f-4330-b0c9-f77078833fd1-kube-api-access-42q7n\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" (UID: \"784b3e4a-c33f-4330-b0c9-f77078833fd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:07.715780 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.715746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42q7n\" (UniqueName: \"kubernetes.io/projected/784b3e4a-c33f-4330-b0c9-f77078833fd1-kube-api-access-42q7n\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" (UID: \"784b3e4a-c33f-4330-b0c9-f77078833fd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:07.715988 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.715801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784b3e4a-c33f-4330-b0c9-f77078833fd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" (UID: \"784b3e4a-c33f-4330-b0c9-f77078833fd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:07.716160 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.716144 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784b3e4a-c33f-4330-b0c9-f77078833fd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" (UID: \"784b3e4a-c33f-4330-b0c9-f77078833fd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:07.725718 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.725687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42q7n\" (UniqueName: \"kubernetes.io/projected/784b3e4a-c33f-4330-b0c9-f77078833fd1-kube-api-access-42q7n\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" (UID: \"784b3e4a-c33f-4330-b0c9-f77078833fd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:07.826229 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:07.826129 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:08.108155 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:08.108043 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w"] Sep 15 11:47:08.111855 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:08.111824 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784b3e4a_c33f_4330_b0c9_f77078833fd1.slice/crio-c205c727fe2316e215d2aed9fe2a337b2099a10101b628840159d0caf894425d WatchSource:0}: Error finding container c205c727fe2316e215d2aed9fe2a337b2099a10101b628840159d0caf894425d: Status 404 returned error can't find the container with id c205c727fe2316e215d2aed9fe2a337b2099a10101b628840159d0caf894425d Sep 15 11:47:08.128658 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:08.128629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" event={"ID":"784b3e4a-c33f-4330-b0c9-f77078833fd1","Type":"ContainerStarted","Data":"c205c727fe2316e215d2aed9fe2a337b2099a10101b628840159d0caf894425d"} Sep 15 11:47:13.099746 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:13.099717 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-znvd5" Sep 15 11:47:16.119120 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:16.119093 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:18.131447 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:18.131422 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-fgnnv" Sep 15 11:47:23.195893 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:23.195854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" event={"ID":"784b3e4a-c33f-4330-b0c9-f77078833fd1","Type":"ContainerStarted","Data":"0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb"} Sep 15 11:47:23.196384 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:23.196028 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:23.216656 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:23.216600 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" podStartSLOduration=1.735618895 podStartE2EDuration="16.216584733s" totalImagesPullingTime="14.480965838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:23.214366962 +0000 UTC m=+530.559475566" watchObservedRunningTime="2026-09-15 11:47:23.216584733 +0000 UTC m=+530.561693337" Sep 15 11:47:25.245875 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.245792 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4"] Sep 15 11:47:25.249167 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.249148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.251476 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.251452 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 15 11:47:25.251593 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.251517 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 15 11:47:25.251593 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.251521 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cvtsx\"" Sep 15 11:47:25.258638 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.258619 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4"] Sep 15 11:47:25.361196 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.361157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8np\" (UniqueName: \"kubernetes.io/projected/35bc74e8-130c-45c7-aec2-c0b72f02fb90-kube-api-access-9x8np\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.361196 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.361224 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/35bc74e8-130c-45c7-aec2-c0b72f02fb90-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.361430 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.361324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35bc74e8-130c-45c7-aec2-c0b72f02fb90-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.462505 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.462463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35bc74e8-130c-45c7-aec2-c0b72f02fb90-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.462647 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.462522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8np\" (UniqueName: \"kubernetes.io/projected/35bc74e8-130c-45c7-aec2-c0b72f02fb90-kube-api-access-9x8np\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.462647 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.462629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/35bc74e8-130c-45c7-aec2-c0b72f02fb90-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.463100 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.463075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/35bc74e8-130c-45c7-aec2-c0b72f02fb90-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.465175 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.465152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/35bc74e8-130c-45c7-aec2-c0b72f02fb90-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.471917 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.471899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8np\" (UniqueName: \"kubernetes.io/projected/35bc74e8-130c-45c7-aec2-c0b72f02fb90-kube-api-access-9x8np\") pod \"kuadrant-console-plugin-fbdd4f445-n28h4\" (UID: \"35bc74e8-130c-45c7-aec2-c0b72f02fb90\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.557995 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.557912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" Sep 15 11:47:25.828939 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:25.828911 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4"] Sep 15 11:47:25.831298 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:25.831269 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35bc74e8_130c_45c7_aec2_c0b72f02fb90.slice/crio-9f8496132f9297ef437c783c4f780d60524df8bc390dc5b6e0b895c8e30ecc34 WatchSource:0}: Error finding container 9f8496132f9297ef437c783c4f780d60524df8bc390dc5b6e0b895c8e30ecc34: Status 404 returned error can't find the container with id 9f8496132f9297ef437c783c4f780d60524df8bc390dc5b6e0b895c8e30ecc34 Sep 15 11:47:26.210108 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:26.210077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" event={"ID":"35bc74e8-130c-45c7-aec2-c0b72f02fb90","Type":"ContainerStarted","Data":"9f8496132f9297ef437c783c4f780d60524df8bc390dc5b6e0b895c8e30ecc34"} Sep 15 11:47:31.234431 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:31.234394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" event={"ID":"35bc74e8-130c-45c7-aec2-c0b72f02fb90","Type":"ContainerStarted","Data":"2c2cc9cb3bd0052d918bd36bcbf9e5f66a39c117bb8c5d6733b1f95f69e23383"} Sep 15 11:47:31.253174 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:31.253122 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n28h4" podStartSLOduration=1.871941637 podStartE2EDuration="6.253078991s" totalImagesPullingTime="4.381137354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:31.252135598 +0000 UTC m=+538.597244203" watchObservedRunningTime="2026-09-15 11:47:31.253078991 +0000 UTC m=+538.598187596" Sep 15 11:47:34.201538 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:34.201507 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:35.191771 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.191742 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2"] Sep 15 11:47:35.199517 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.199496 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:35.207283 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.207260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2"] Sep 15 11:47:35.343908 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.343874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk84q\" (UniqueName: \"kubernetes.io/projected/008cab0b-4987-45ee-a9ec-a239b8a66a40-kube-api-access-vk84q\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ghtf2\" (UID: \"008cab0b-4987-45ee-a9ec-a239b8a66a40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:35.344089 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.343925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/008cab0b-4987-45ee-a9ec-a239b8a66a40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ghtf2\" (UID: \"008cab0b-4987-45ee-a9ec-a239b8a66a40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:35.444669 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.444586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk84q\" (UniqueName: \"kubernetes.io/projected/008cab0b-4987-45ee-a9ec-a239b8a66a40-kube-api-access-vk84q\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ghtf2\" (UID: \"008cab0b-4987-45ee-a9ec-a239b8a66a40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:35.444669 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.444630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/008cab0b-4987-45ee-a9ec-a239b8a66a40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ghtf2\" (UID: \"008cab0b-4987-45ee-a9ec-a239b8a66a40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:35.444950 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.444936 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/008cab0b-4987-45ee-a9ec-a239b8a66a40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ghtf2\" (UID: \"008cab0b-4987-45ee-a9ec-a239b8a66a40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:35.459839 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.459806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk84q\" (UniqueName: \"kubernetes.io/projected/008cab0b-4987-45ee-a9ec-a239b8a66a40-kube-api-access-vk84q\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-ghtf2\" (UID: \"008cab0b-4987-45ee-a9ec-a239b8a66a40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:35.509523 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.509489 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:35.805775 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:35.805753 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2"] Sep 15 11:47:35.807935 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:35.807910 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008cab0b_4987_45ee_a9ec_a239b8a66a40.slice/crio-0a69637554be0263637035dc82a4f3c6230c8e72f1633f45161b347a5cb28c5a WatchSource:0}: Error finding container 0a69637554be0263637035dc82a4f3c6230c8e72f1633f45161b347a5cb28c5a: Status 404 returned error can't find the container with id 0a69637554be0263637035dc82a4f3c6230c8e72f1633f45161b347a5cb28c5a Sep 15 11:47:36.257628 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.257600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" event={"ID":"008cab0b-4987-45ee-a9ec-a239b8a66a40","Type":"ContainerStarted","Data":"298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42"} Sep 15 11:47:36.258012 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.257646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" event={"ID":"008cab0b-4987-45ee-a9ec-a239b8a66a40","Type":"ContainerStarted","Data":"0a69637554be0263637035dc82a4f3c6230c8e72f1633f45161b347a5cb28c5a"} Sep 15 11:47:36.258012 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.257706 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:36.279565 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.279519 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" podStartSLOduration=1.279502678 podStartE2EDuration="1.279502678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:36.277259244 +0000 UTC m=+543.622367848" watchObservedRunningTime="2026-09-15 11:47:36.279502678 +0000 UTC m=+543.624611282" Sep 15 11:47:36.339326 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.339288 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2"] Sep 15 11:47:36.349370 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.349339 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2"] Sep 15 11:47:36.367620 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.367593 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w"] Sep 15 11:47:36.367803 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.367779 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" containerName="manager" containerID="cri-o://0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb" gracePeriod=2 Sep 15 11:47:36.368859 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.368799 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq"] Sep 15 11:47:36.369098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.369085 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="008cab0b-4987-45ee-a9ec-a239b8a66a40" containerName="manager" Sep 15 11:47:36.369098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.369099 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="008cab0b-4987-45ee-a9ec-a239b8a66a40" containerName="manager" Sep 15 11:47:36.369230 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.369152 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="008cab0b-4987-45ee-a9ec-a239b8a66a40" containerName="manager" Sep 15 11:47:36.371013 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.370993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:36.382461 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.382437 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w"] Sep 15 11:47:36.391470 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.391404 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"784b3e4a-c33f-4330-b0c9-f77078833fd1\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-mx76w\": pods \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:36.392962 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.392939 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq"] Sep 15 11:47:36.393675 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.393584 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:36.401072 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.400412 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt"] Sep 15 11:47:36.401072 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.400855 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" containerName="manager" Sep 15 11:47:36.401072 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.400871 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" containerName="manager" Sep 15 11:47:36.401072 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.400943 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" containerName="manager" Sep 15 11:47:36.405129 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.404891 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn"] Sep 15 11:47:36.405129 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.405076 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" containerName="manager" containerID="cri-o://ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85" gracePeriod=2 Sep 15 11:47:36.405828 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.405479 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:36.407395 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.407372 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn"] Sep 15 11:47:36.417193 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.417164 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt"] Sep 15 11:47:36.423100 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.423075 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc"] Sep 15 11:47:36.423394 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.423382 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" containerName="manager" Sep 15 11:47:36.423442 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.423396 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" containerName="manager" Sep 15 11:47:36.423474 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.423450 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" containerName="manager" Sep 15 11:47:36.425576 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.425563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" Sep 15 11:47:36.431744 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.431718 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:36.433416 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.433388 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rr4hn\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:36.435082 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.435056 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:36.436832 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.436794 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rr4hn\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:36.437596 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.437575 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc"] Sep 15 11:47:36.453160 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.453132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vzf\" (UniqueName: \"kubernetes.io/projected/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-kube-api-access-58vzf\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-g8jvq\" (UID: \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:36.453339 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.453171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-g8jvq\" (UID: \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:36.554316 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.554282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bct\" (UniqueName: \"kubernetes.io/projected/39e7a5ff-de53-4728-81a0-d9f795c4fa47-kube-api-access-57bct\") pod \"limitador-operator-controller-manager-74495c69f8-clmqc\" (UID: \"39e7a5ff-de53-4728-81a0-d9f795c4fa47\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" Sep 15 11:47:36.554466 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.554325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-g8jvq\" (UID: \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:36.554466 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.554359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m2vl\" (UniqueName: \"kubernetes.io/projected/3a8cfe38-aa81-4f99-9077-2413efaa2129-kube-api-access-4m2vl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k42zt\" (UID: \"3a8cfe38-aa81-4f99-9077-2413efaa2129\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:36.554539 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.554471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a8cfe38-aa81-4f99-9077-2413efaa2129-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k42zt\" (UID: \"3a8cfe38-aa81-4f99-9077-2413efaa2129\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:36.554571 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.554540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vzf\" (UniqueName: \"kubernetes.io/projected/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-kube-api-access-58vzf\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-g8jvq\" (UID: \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:36.554700 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.554685 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-g8jvq\" (UID: \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:36.567092 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.567058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vzf\" (UniqueName: \"kubernetes.io/projected/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-kube-api-access-58vzf\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-g8jvq\" (UID: \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:36.654934 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.654912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a8cfe38-aa81-4f99-9077-2413efaa2129-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k42zt\" (UID: \"3a8cfe38-aa81-4f99-9077-2413efaa2129\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:36.655065 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.654963 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bct\" (UniqueName: \"kubernetes.io/projected/39e7a5ff-de53-4728-81a0-d9f795c4fa47-kube-api-access-57bct\") pod \"limitador-operator-controller-manager-74495c69f8-clmqc\" (UID: \"39e7a5ff-de53-4728-81a0-d9f795c4fa47\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" Sep 15 11:47:36.655065 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.654994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m2vl\" (UniqueName: \"kubernetes.io/projected/3a8cfe38-aa81-4f99-9077-2413efaa2129-kube-api-access-4m2vl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k42zt\" (UID: \"3a8cfe38-aa81-4f99-9077-2413efaa2129\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:36.655388 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.655365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a8cfe38-aa81-4f99-9077-2413efaa2129-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k42zt\" (UID: \"3a8cfe38-aa81-4f99-9077-2413efaa2129\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:36.655429 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.655413 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:36.657315 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.657290 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:36.658318 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.658301 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:36.659001 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.658979 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rr4hn\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:36.660711 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.660690 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mx76w\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:36.662488 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:36.662460 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-rr4hn\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:36.664174 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.664155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m2vl\" (UniqueName: \"kubernetes.io/projected/3a8cfe38-aa81-4f99-9077-2413efaa2129-kube-api-access-4m2vl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-k42zt\" (UID: \"3a8cfe38-aa81-4f99-9077-2413efaa2129\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:36.664487 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.664470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bct\" (UniqueName: \"kubernetes.io/projected/39e7a5ff-de53-4728-81a0-d9f795c4fa47-kube-api-access-57bct\") pod \"limitador-operator-controller-manager-74495c69f8-clmqc\" (UID: \"39e7a5ff-de53-4728-81a0-d9f795c4fa47\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" Sep 15 11:47:36.682820 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.682794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:36.756022 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.755993 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b929b\" (UniqueName: \"kubernetes.io/projected/67b8f7bf-551b-4e8a-8ca1-574d8ff17436-kube-api-access-b929b\") pod \"67b8f7bf-551b-4e8a-8ca1-574d8ff17436\" (UID: \"67b8f7bf-551b-4e8a-8ca1-574d8ff17436\") " Sep 15 11:47:36.756165 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.756066 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784b3e4a-c33f-4330-b0c9-f77078833fd1-extensions-socket-volume\") pod \"784b3e4a-c33f-4330-b0c9-f77078833fd1\" (UID: \"784b3e4a-c33f-4330-b0c9-f77078833fd1\") " Sep 15 11:47:36.756248 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.756228 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42q7n\" (UniqueName: \"kubernetes.io/projected/784b3e4a-c33f-4330-b0c9-f77078833fd1-kube-api-access-42q7n\") pod \"784b3e4a-c33f-4330-b0c9-f77078833fd1\" (UID: \"784b3e4a-c33f-4330-b0c9-f77078833fd1\") " Sep 15 11:47:36.756398 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.756379 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784b3e4a-c33f-4330-b0c9-f77078833fd1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "784b3e4a-c33f-4330-b0c9-f77078833fd1" (UID: "784b3e4a-c33f-4330-b0c9-f77078833fd1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:47:36.756476 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.756468 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784b3e4a-c33f-4330-b0c9-f77078833fd1-extensions-socket-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:47:36.758285 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.758257 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b8f7bf-551b-4e8a-8ca1-574d8ff17436-kube-api-access-b929b" (OuterVolumeSpecName: "kube-api-access-b929b") pod "67b8f7bf-551b-4e8a-8ca1-574d8ff17436" (UID: "67b8f7bf-551b-4e8a-8ca1-574d8ff17436"). InnerVolumeSpecName "kube-api-access-b929b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:47:36.758381 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.758314 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784b3e4a-c33f-4330-b0c9-f77078833fd1-kube-api-access-42q7n" (OuterVolumeSpecName: "kube-api-access-42q7n") pod "784b3e4a-c33f-4330-b0c9-f77078833fd1" (UID: "784b3e4a-c33f-4330-b0c9-f77078833fd1"). InnerVolumeSpecName "kube-api-access-42q7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:47:36.788629 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.788560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:36.795093 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.795069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" Sep 15 11:47:36.857868 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.857836 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b929b\" (UniqueName: \"kubernetes.io/projected/67b8f7bf-551b-4e8a-8ca1-574d8ff17436-kube-api-access-b929b\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:47:36.857982 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.857875 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42q7n\" (UniqueName: \"kubernetes.io/projected/784b3e4a-c33f-4330-b0c9-f77078833fd1-kube-api-access-42q7n\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:47:36.961786 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:36.961761 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq"] Sep 15 11:47:36.964546 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:36.964523 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318b4d6e_e956_4f72_9ad1_c43b6ad6db1c.slice/crio-5638d60a4fea2e43f6c9eb71072d56d5d9a6016befff31c54a330053ead14f81 WatchSource:0}: Error finding container 5638d60a4fea2e43f6c9eb71072d56d5d9a6016befff31c54a330053ead14f81: Status 404 returned error can't find the container with id 5638d60a4fea2e43f6c9eb71072d56d5d9a6016befff31c54a330053ead14f81 Sep 15 11:47:37.064671 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.064600 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt"] Sep 15 11:47:37.068091 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:37.068056 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8cfe38_aa81_4f99_9077_2413efaa2129.slice/crio-507d9a46141d916b5a14d3b9d1a5cf21b59fe6355590daddc92af537bb76d192 WatchSource:0}: Error finding container 507d9a46141d916b5a14d3b9d1a5cf21b59fe6355590daddc92af537bb76d192: Status 404 returned error can't find the container with id 507d9a46141d916b5a14d3b9d1a5cf21b59fe6355590daddc92af537bb76d192 Sep 15 11:47:37.074075 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.074053 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc"] Sep 15 11:47:37.076722 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:37.076697 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e7a5ff_de53_4728_81a0_d9f795c4fa47.slice/crio-eea4fbedde886770cc5f03312dbdbf0d89015ca81fbcd4058af95b97d29c3850 WatchSource:0}: Error finding container eea4fbedde886770cc5f03312dbdbf0d89015ca81fbcd4058af95b97d29c3850: Status 404 returned error can't find the container with id eea4fbedde886770cc5f03312dbdbf0d89015ca81fbcd4058af95b97d29c3850 Sep 15 11:47:37.187582 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.187552 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" path="/var/lib/kubelet/pods/67b8f7bf-551b-4e8a-8ca1-574d8ff17436/volumes" Sep 15 11:47:37.187891 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.187879 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784b3e4a-c33f-4330-b0c9-f77078833fd1" path="/var/lib/kubelet/pods/784b3e4a-c33f-4330-b0c9-f77078833fd1/volumes" Sep 15 11:47:37.263731 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.263695 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" event={"ID":"39e7a5ff-de53-4728-81a0-d9f795c4fa47","Type":"ContainerStarted","Data":"740234e29c3392dc9ae42226358171902c5341e6dda50d064832abc58bb8bb3e"} Sep 15 11:47:37.264173 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.263754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" event={"ID":"39e7a5ff-de53-4728-81a0-d9f795c4fa47","Type":"ContainerStarted","Data":"eea4fbedde886770cc5f03312dbdbf0d89015ca81fbcd4058af95b97d29c3850"} Sep 15 11:47:37.264173 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.263850 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" Sep 15 11:47:37.265027 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.265006 2579 generic.go:356] "Generic (PLEG): container finished" podID="784b3e4a-c33f-4330-b0c9-f77078833fd1" containerID="0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb" exitCode=0 Sep 15 11:47:37.265092 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.265036 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mx76w" Sep 15 11:47:37.265150 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.265098 2579 scope.go:117] "RemoveContainer" containerID="0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb" Sep 15 11:47:37.267278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.266876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" event={"ID":"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c","Type":"ContainerStarted","Data":"61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255"} Sep 15 11:47:37.267278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.266922 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" event={"ID":"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c","Type":"ContainerStarted","Data":"5638d60a4fea2e43f6c9eb71072d56d5d9a6016befff31c54a330053ead14f81"} Sep 15 11:47:37.267278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.266953 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:37.268617 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.268594 2579 generic.go:356] "Generic (PLEG): container finished" podID="67b8f7bf-551b-4e8a-8ca1-574d8ff17436" containerID="ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85" exitCode=0 Sep 15 11:47:37.268721 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.268625 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rr4hn" Sep 15 11:47:37.270432 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.270408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" event={"ID":"3a8cfe38-aa81-4f99-9077-2413efaa2129","Type":"ContainerStarted","Data":"bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd"} Sep 15 11:47:37.270524 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.270453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" event={"ID":"3a8cfe38-aa81-4f99-9077-2413efaa2129","Type":"ContainerStarted","Data":"507d9a46141d916b5a14d3b9d1a5cf21b59fe6355590daddc92af537bb76d192"} Sep 15 11:47:37.270579 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:37.270534 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" container="manager" Sep 15 11:47:37.274401 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.274386 2579 scope.go:117] "RemoveContainer" containerID="0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb" Sep 15 11:47:37.274696 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:37.274665 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb\": container with ID starting with 0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb not found: ID does not exist" containerID="0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb" Sep 15 11:47:37.274799 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.274697 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb"} err="failed to get container status \"0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb\": rpc error: code = NotFound desc = could not find container \"0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb\": container with ID starting with 0395cfe07d0f91e51eac873dd3de206908dfacac6edb3d460c469e85164510cb not found: ID does not exist" Sep 15 11:47:37.274799 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.274718 2579 scope.go:117] "RemoveContainer" containerID="ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85" Sep 15 11:47:37.283097 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.283079 2579 scope.go:117] "RemoveContainer" containerID="ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85" Sep 15 11:47:37.283415 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:37.283397 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85\": container with ID starting with ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85 not found: ID does not exist" containerID="ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85" Sep 15 11:47:37.283516 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.283420 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85"} err="failed to get container status \"ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85\": rpc error: code = NotFound desc = could not find container \"ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85\": container with ID starting with ebe013c4d1adc65991f3b560ba77446c803f1005f1d3a0ca9ee4d1a94c118d85 not found: ID does not exist" Sep 15 11:47:37.284278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.284174 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" podStartSLOduration=1.284160892 podStartE2EDuration="1.284160892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:37.282421113 +0000 UTC m=+544.627529717" watchObservedRunningTime="2026-09-15 11:47:37.284160892 +0000 UTC m=+544.629269496" Sep 15 11:47:37.305261 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.305222 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" podStartSLOduration=1.305185948 podStartE2EDuration="1.305185948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:37.304352286 +0000 UTC m=+544.649460892" watchObservedRunningTime="2026-09-15 11:47:37.305185948 +0000 UTC m=+544.650294552" Sep 15 11:47:37.330869 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:37.330784 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" podStartSLOduration=1.330770108 podStartE2EDuration="1.330770108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:37.3294802 +0000 UTC m=+544.674588806" watchObservedRunningTime="2026-09-15 11:47:37.330770108 +0000 UTC m=+544.675878801" Sep 15 11:47:38.277947 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.277899 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" podUID="008cab0b-4987-45ee-a9ec-a239b8a66a40" containerName="manager" containerID="cri-o://298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42" gracePeriod=2 Sep 15 11:47:38.279277 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.279250 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:38.507332 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.507308 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:38.509420 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:38.509396 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-ghtf2\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="008cab0b-4987-45ee-a9ec-a239b8a66a40" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:38.672503 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.672417 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/008cab0b-4987-45ee-a9ec-a239b8a66a40-extensions-socket-volume\") pod \"008cab0b-4987-45ee-a9ec-a239b8a66a40\" (UID: \"008cab0b-4987-45ee-a9ec-a239b8a66a40\") " Sep 15 11:47:38.672503 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.672497 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk84q\" (UniqueName: \"kubernetes.io/projected/008cab0b-4987-45ee-a9ec-a239b8a66a40-kube-api-access-vk84q\") pod \"008cab0b-4987-45ee-a9ec-a239b8a66a40\" (UID: \"008cab0b-4987-45ee-a9ec-a239b8a66a40\") " Sep 15 11:47:38.672805 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.672781 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008cab0b-4987-45ee-a9ec-a239b8a66a40-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "008cab0b-4987-45ee-a9ec-a239b8a66a40" (UID: "008cab0b-4987-45ee-a9ec-a239b8a66a40"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:47:38.674638 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.674618 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008cab0b-4987-45ee-a9ec-a239b8a66a40-kube-api-access-vk84q" (OuterVolumeSpecName: "kube-api-access-vk84q") pod "008cab0b-4987-45ee-a9ec-a239b8a66a40" (UID: "008cab0b-4987-45ee-a9ec-a239b8a66a40"). InnerVolumeSpecName "kube-api-access-vk84q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:47:38.773739 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.773698 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/008cab0b-4987-45ee-a9ec-a239b8a66a40-extensions-socket-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:47:38.773739 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:38.773731 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk84q\" (UniqueName: \"kubernetes.io/projected/008cab0b-4987-45ee-a9ec-a239b8a66a40-kube-api-access-vk84q\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:47:39.188076 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:39.188036 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="008cab0b-4987-45ee-a9ec-a239b8a66a40" path="/var/lib/kubelet/pods/008cab0b-4987-45ee-a9ec-a239b8a66a40/volumes" Sep 15 11:47:39.290535 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:39.290499 2579 generic.go:356] "Generic (PLEG): container finished" podID="008cab0b-4987-45ee-a9ec-a239b8a66a40" containerID="298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42" exitCode=0 Sep 15 11:47:39.290927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:39.290552 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-ghtf2" Sep 15 11:47:39.290927 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:39.290593 2579 scope.go:117] "RemoveContainer" containerID="298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42" Sep 15 11:47:39.298727 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:39.298710 2579 scope.go:117] "RemoveContainer" containerID="298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42" Sep 15 11:47:39.299024 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:39.299005 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42\": container with ID starting with 298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42 not found: ID does not exist" containerID="298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42" Sep 15 11:47:39.299095 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:39.299030 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42"} err="failed to get container status \"298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42\": rpc error: code = NotFound desc = could not find container \"298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42\": container with ID starting with 298609e138026c9e3fb635dbe6850c1a681d80bea6876297b0ae9e9d5d510d42 not found: ID does not exist" Sep 15 11:47:48.281100 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.281068 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-clmqc" Sep 15 11:47:48.281555 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.281144 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:47:48.354386 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.354355 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt"] Sep 15 11:47:48.354595 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.354568 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" podUID="3a8cfe38-aa81-4f99-9077-2413efaa2129" containerName="manager" containerID="cri-o://bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd" gracePeriod=10 Sep 15 11:47:48.356312 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.356286 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:48.603082 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.603058 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:48.634812 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.634784 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8"] Sep 15 11:47:48.635136 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.635119 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8cfe38-aa81-4f99-9077-2413efaa2129" containerName="manager" Sep 15 11:47:48.635261 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.635139 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8cfe38-aa81-4f99-9077-2413efaa2129" containerName="manager" Sep 15 11:47:48.635261 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.635240 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a8cfe38-aa81-4f99-9077-2413efaa2129" containerName="manager" Sep 15 11:47:48.637230 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.637194 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:48.658803 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.658772 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8"] Sep 15 11:47:48.759131 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.759096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m2vl\" (UniqueName: \"kubernetes.io/projected/3a8cfe38-aa81-4f99-9077-2413efaa2129-kube-api-access-4m2vl\") pod \"3a8cfe38-aa81-4f99-9077-2413efaa2129\" (UID: \"3a8cfe38-aa81-4f99-9077-2413efaa2129\") " Sep 15 11:47:48.759298 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.759154 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a8cfe38-aa81-4f99-9077-2413efaa2129-extensions-socket-volume\") pod \"3a8cfe38-aa81-4f99-9077-2413efaa2129\" (UID: \"3a8cfe38-aa81-4f99-9077-2413efaa2129\") " Sep 15 11:47:48.759362 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.759306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwwsq\" (UniqueName: \"kubernetes.io/projected/1b8589f0-8c9f-47eb-ba14-4e9578f13774-kube-api-access-kwwsq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6nb8\" (UID: \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:48.759362 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.759337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b8589f0-8c9f-47eb-ba14-4e9578f13774-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6nb8\" (UID: \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:48.759576 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.759548 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8cfe38-aa81-4f99-9077-2413efaa2129-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3a8cfe38-aa81-4f99-9077-2413efaa2129" (UID: "3a8cfe38-aa81-4f99-9077-2413efaa2129"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:47:48.761360 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.761336 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8cfe38-aa81-4f99-9077-2413efaa2129-kube-api-access-4m2vl" (OuterVolumeSpecName: "kube-api-access-4m2vl") pod "3a8cfe38-aa81-4f99-9077-2413efaa2129" (UID: "3a8cfe38-aa81-4f99-9077-2413efaa2129"). InnerVolumeSpecName "kube-api-access-4m2vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:47:48.860470 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.860376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwwsq\" (UniqueName: \"kubernetes.io/projected/1b8589f0-8c9f-47eb-ba14-4e9578f13774-kube-api-access-kwwsq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6nb8\" (UID: \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:48.860470 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.860420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b8589f0-8c9f-47eb-ba14-4e9578f13774-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6nb8\" (UID: \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:48.860651 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.860497 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m2vl\" (UniqueName: \"kubernetes.io/projected/3a8cfe38-aa81-4f99-9077-2413efaa2129-kube-api-access-4m2vl\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:47:48.860651 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.860507 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a8cfe38-aa81-4f99-9077-2413efaa2129-extensions-socket-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:47:48.860818 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.860803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b8589f0-8c9f-47eb-ba14-4e9578f13774-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6nb8\" (UID: \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:48.877500 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.877476 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwwsq\" (UniqueName: \"kubernetes.io/projected/1b8589f0-8c9f-47eb-ba14-4e9578f13774-kube-api-access-kwwsq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-b6nb8\" (UID: \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:48.951063 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:48.951028 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:49.225392 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.225280 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8"] Sep 15 11:47:49.228270 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:47:49.228250 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8589f0_8c9f_47eb_ba14_4e9578f13774.slice/crio-40eee8a2dd7b653c75511ed30e0d6c840181afbf5e28d5719548ee8f41095001 WatchSource:0}: Error finding container 40eee8a2dd7b653c75511ed30e0d6c840181afbf5e28d5719548ee8f41095001: Status 404 returned error can't find the container with id 40eee8a2dd7b653c75511ed30e0d6c840181afbf5e28d5719548ee8f41095001 Sep 15 11:47:49.345595 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.345556 2579 generic.go:356] "Generic (PLEG): container finished" podID="3a8cfe38-aa81-4f99-9077-2413efaa2129" containerID="bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd" exitCode=0 Sep 15 11:47:49.345595 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.345596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" event={"ID":"3a8cfe38-aa81-4f99-9077-2413efaa2129","Type":"ContainerDied","Data":"bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd"} Sep 15 11:47:49.346087 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.345613 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" Sep 15 11:47:49.346087 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.345682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt" event={"ID":"3a8cfe38-aa81-4f99-9077-2413efaa2129","Type":"ContainerDied","Data":"507d9a46141d916b5a14d3b9d1a5cf21b59fe6355590daddc92af537bb76d192"} Sep 15 11:47:49.346087 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.345714 2579 scope.go:117] "RemoveContainer" containerID="bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd" Sep 15 11:47:49.348085 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.348040 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" event={"ID":"1b8589f0-8c9f-47eb-ba14-4e9578f13774","Type":"ContainerStarted","Data":"c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0"} Sep 15 11:47:49.348193 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.348096 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" event={"ID":"1b8589f0-8c9f-47eb-ba14-4e9578f13774","Type":"ContainerStarted","Data":"40eee8a2dd7b653c75511ed30e0d6c840181afbf5e28d5719548ee8f41095001"} Sep 15 11:47:49.348295 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.348193 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:47:49.354300 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.354277 2579 scope.go:117] "RemoveContainer" containerID="bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd" Sep 15 11:47:49.354642 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:47:49.354622 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd\": container with ID starting with bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd not found: ID does not exist" containerID="bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd" Sep 15 11:47:49.354720 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.354651 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd"} err="failed to get container status \"bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd\": rpc error: code = NotFound desc = could not find container \"bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd\": container with ID starting with bf6fe51f15618085e5de7024dcf3065495aca44539c4946e095f66ea868e4dcd not found: ID does not exist" Sep 15 11:47:49.374738 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.374695 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" podStartSLOduration=1.37468085 podStartE2EDuration="1.37468085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:49.374121488 +0000 UTC m=+556.719230107" watchObservedRunningTime="2026-09-15 11:47:49.37468085 +0000 UTC m=+556.719789454" Sep 15 11:47:49.390973 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.390944 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt"] Sep 15 11:47:49.401923 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:49.401897 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-k42zt"] Sep 15 11:47:51.186878 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:47:51.186834 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a8cfe38-aa81-4f99-9077-2413efaa2129" path="/var/lib/kubelet/pods/3a8cfe38-aa81-4f99-9077-2413efaa2129/volumes" Sep 15 11:48:00.356144 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.356108 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 11:48:00.411415 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.411383 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq"] Sep 15 11:48:00.411627 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.411587 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" podUID="318b4d6e-e956-4f72-9ad1-c43b6ad6db1c" containerName="manager" containerID="cri-o://61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255" gracePeriod=10 Sep 15 11:48:00.666361 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.666339 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:48:00.754608 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.754577 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58vzf\" (UniqueName: \"kubernetes.io/projected/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-kube-api-access-58vzf\") pod \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\" (UID: \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\") " Sep 15 11:48:00.754772 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.754671 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-extensions-socket-volume\") pod \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\" (UID: \"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c\") " Sep 15 11:48:00.755045 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.755015 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "318b4d6e-e956-4f72-9ad1-c43b6ad6db1c" (UID: "318b4d6e-e956-4f72-9ad1-c43b6ad6db1c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 11:48:00.756685 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.756657 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-kube-api-access-58vzf" (OuterVolumeSpecName: "kube-api-access-58vzf") pod "318b4d6e-e956-4f72-9ad1-c43b6ad6db1c" (UID: "318b4d6e-e956-4f72-9ad1-c43b6ad6db1c"). InnerVolumeSpecName "kube-api-access-58vzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:48:00.855287 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.855253 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-extensions-socket-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:48:00.855287 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:00.855282 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58vzf\" (UniqueName: \"kubernetes.io/projected/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c-kube-api-access-58vzf\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:48:01.405887 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.405855 2579 generic.go:356] "Generic (PLEG): container finished" podID="318b4d6e-e956-4f72-9ad1-c43b6ad6db1c" containerID="61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255" exitCode=0 Sep 15 11:48:01.406299 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.405905 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" Sep 15 11:48:01.406299 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.405916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" event={"ID":"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c","Type":"ContainerDied","Data":"61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255"} Sep 15 11:48:01.406299 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.405957 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq" event={"ID":"318b4d6e-e956-4f72-9ad1-c43b6ad6db1c","Type":"ContainerDied","Data":"5638d60a4fea2e43f6c9eb71072d56d5d9a6016befff31c54a330053ead14f81"} Sep 15 11:48:01.406299 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.405977 2579 scope.go:117] "RemoveContainer" containerID="61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255" Sep 15 11:48:01.414366 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.414117 2579 scope.go:117] "RemoveContainer" containerID="61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255" Sep 15 11:48:01.414442 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:48:01.414422 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255\": container with ID starting with 61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255 not found: ID does not exist" containerID="61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255" Sep 15 11:48:01.414480 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.414449 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255"} err="failed to get container status \"61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255\": rpc error: code = NotFound desc = could not find container \"61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255\": container with ID starting with 61304b6dc277f7123a18672889c7590d57165f69ae49cef40f91f67b3f03f255 not found: ID does not exist" Sep 15 11:48:01.425872 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.425848 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq"] Sep 15 11:48:01.432883 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:01.432855 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-g8jvq"] Sep 15 11:48:03.187971 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:03.187926 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="318b4d6e-e956-4f72-9ad1-c43b6ad6db1c" path="/var/lib/kubelet/pods/318b4d6e-e956-4f72-9ad1-c43b6ad6db1c/volumes" Sep 15 11:48:17.290829 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.290797 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t5tp6"] Sep 15 11:48:17.291186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.291119 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="318b4d6e-e956-4f72-9ad1-c43b6ad6db1c" containerName="manager" Sep 15 11:48:17.291186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.291129 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="318b4d6e-e956-4f72-9ad1-c43b6ad6db1c" containerName="manager" Sep 15 11:48:17.291186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.291180 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="318b4d6e-e956-4f72-9ad1-c43b6ad6db1c" containerName="manager" Sep 15 11:48:17.305705 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.305674 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t5tp6"] Sep 15 11:48:17.305857 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.305784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:17.308135 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.308099 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 15 11:48:17.382357 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.382323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t5tp6"] Sep 15 11:48:17.388108 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.388081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvnl\" (UniqueName: \"kubernetes.io/projected/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-kube-api-access-qjvnl\") pod \"limitador-limitador-8588749977-t5tp6\" (UID: \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\") " pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:17.388278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.388145 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-config-file\") pod \"limitador-limitador-8588749977-t5tp6\" (UID: \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\") " pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:17.489110 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.489074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-config-file\") pod \"limitador-limitador-8588749977-t5tp6\" (UID: \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\") " pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:17.489339 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.489142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvnl\" (UniqueName: \"kubernetes.io/projected/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-kube-api-access-qjvnl\") pod \"limitador-limitador-8588749977-t5tp6\" (UID: \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\") " pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:17.489843 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.489820 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-config-file\") pod \"limitador-limitador-8588749977-t5tp6\" (UID: \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\") " pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:17.498392 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.498360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvnl\" (UniqueName: \"kubernetes.io/projected/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-kube-api-access-qjvnl\") pod \"limitador-limitador-8588749977-t5tp6\" (UID: \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\") " pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:17.615788 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.615702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:17.889935 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:17.889912 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t5tp6"] Sep 15 11:48:17.892561 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:48:17.892533 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afbe7a3_165b_461b_a1d4_8b16f8b2a6c9.slice/crio-be1e443e80609ac338a0f401475b835e5c95721bbe8b7cf8fa2351942fa4436f WatchSource:0}: Error finding container be1e443e80609ac338a0f401475b835e5c95721bbe8b7cf8fa2351942fa4436f: Status 404 returned error can't find the container with id be1e443e80609ac338a0f401475b835e5c95721bbe8b7cf8fa2351942fa4436f Sep 15 11:48:18.124914 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.124883 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-br2fj"] Sep 15 11:48:18.129604 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.129582 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" Sep 15 11:48:18.131678 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.131656 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5lr4b\"" Sep 15 11:48:18.135757 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.135737 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-br2fj"] Sep 15 11:48:18.194539 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.194506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6lrw\" (UniqueName: \"kubernetes.io/projected/d85a033b-5293-4480-be91-fab88f9268cc-kube-api-access-g6lrw\") pod \"authorino-7bcfd8f9cd-br2fj\" (UID: \"d85a033b-5293-4480-be91-fab88f9268cc\") " pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" Sep 15 11:48:18.295410 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.295379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6lrw\" (UniqueName: \"kubernetes.io/projected/d85a033b-5293-4480-be91-fab88f9268cc-kube-api-access-g6lrw\") pod \"authorino-7bcfd8f9cd-br2fj\" (UID: \"d85a033b-5293-4480-be91-fab88f9268cc\") " pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" Sep 15 11:48:18.305492 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.305466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6lrw\" (UniqueName: \"kubernetes.io/projected/d85a033b-5293-4480-be91-fab88f9268cc-kube-api-access-g6lrw\") pod \"authorino-7bcfd8f9cd-br2fj\" (UID: \"d85a033b-5293-4480-be91-fab88f9268cc\") " pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" Sep 15 11:48:18.440862 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.440830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" Sep 15 11:48:18.490443 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.490372 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" event={"ID":"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9","Type":"ContainerStarted","Data":"be1e443e80609ac338a0f401475b835e5c95721bbe8b7cf8fa2351942fa4436f"} Sep 15 11:48:18.709464 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:18.709388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-br2fj"] Sep 15 11:48:18.713432 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:48:18.713400 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd85a033b_5293_4480_be91_fab88f9268cc.slice/crio-e636bf2eabedd45da4ecd4ea11d8cff99cb9262bb304d7c54a7b50702c67f270 WatchSource:0}: Error finding container e636bf2eabedd45da4ecd4ea11d8cff99cb9262bb304d7c54a7b50702c67f270: Status 404 returned error can't find the container with id e636bf2eabedd45da4ecd4ea11d8cff99cb9262bb304d7c54a7b50702c67f270 Sep 15 11:48:19.497508 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:19.497473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" event={"ID":"d85a033b-5293-4480-be91-fab88f9268cc","Type":"ContainerStarted","Data":"e636bf2eabedd45da4ecd4ea11d8cff99cb9262bb304d7c54a7b50702c67f270"} Sep 15 11:48:19.499124 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:19.499073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" event={"ID":"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9","Type":"ContainerStarted","Data":"8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e"} Sep 15 11:48:19.499327 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:19.499212 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:19.517028 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:19.516985 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" podStartSLOduration=1.332662631 podStartE2EDuration="2.516969102s" totalImagesPullingTime="1.184306471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:19.515775407 +0000 UTC m=+586.860884014" watchObservedRunningTime="2026-09-15 11:48:19.516969102 +0000 UTC m=+586.862077732" Sep 15 11:48:20.506532 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:20.506499 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" event={"ID":"d85a033b-5293-4480-be91-fab88f9268cc","Type":"ContainerStarted","Data":"c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634"} Sep 15 11:48:20.533135 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:20.533070 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" podStartSLOduration=0.859652844 podStartE2EDuration="2.533049986s" totalImagesPullingTime="1.673397142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:20.526425784 +0000 UTC m=+587.871534390" watchObservedRunningTime="2026-09-15 11:48:20.533049986 +0000 UTC m=+587.878158593" Sep 15 11:48:22.481273 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:22.481243 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-br2fj"] Sep 15 11:48:22.517049 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:22.516989 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" podUID="d85a033b-5293-4480-be91-fab88f9268cc" containerName="authorino" containerID="cri-o://c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634" gracePeriod=30 Sep 15 11:48:22.768564 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:22.768542 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" Sep 15 11:48:22.833825 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:22.833795 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6lrw\" (UniqueName: \"kubernetes.io/projected/d85a033b-5293-4480-be91-fab88f9268cc-kube-api-access-g6lrw\") pod \"d85a033b-5293-4480-be91-fab88f9268cc\" (UID: \"d85a033b-5293-4480-be91-fab88f9268cc\") " Sep 15 11:48:22.835911 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:22.835887 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85a033b-5293-4480-be91-fab88f9268cc-kube-api-access-g6lrw" (OuterVolumeSpecName: "kube-api-access-g6lrw") pod "d85a033b-5293-4480-be91-fab88f9268cc" (UID: "d85a033b-5293-4480-be91-fab88f9268cc"). InnerVolumeSpecName "kube-api-access-g6lrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:48:22.935237 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:22.935188 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6lrw\" (UniqueName: \"kubernetes.io/projected/d85a033b-5293-4480-be91-fab88f9268cc-kube-api-access-g6lrw\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:48:23.522685 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.522603 2579 generic.go:356] "Generic (PLEG): container finished" podID="d85a033b-5293-4480-be91-fab88f9268cc" containerID="c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634" exitCode=0 Sep 15 11:48:23.522685 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.522644 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" Sep 15 11:48:23.523124 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.522692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" event={"ID":"d85a033b-5293-4480-be91-fab88f9268cc","Type":"ContainerDied","Data":"c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634"} Sep 15 11:48:23.523124 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.522734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-br2fj" event={"ID":"d85a033b-5293-4480-be91-fab88f9268cc","Type":"ContainerDied","Data":"e636bf2eabedd45da4ecd4ea11d8cff99cb9262bb304d7c54a7b50702c67f270"} Sep 15 11:48:23.523124 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.522753 2579 scope.go:117] "RemoveContainer" containerID="c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634" Sep 15 11:48:23.530973 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.530954 2579 scope.go:117] "RemoveContainer" containerID="c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634" Sep 15 11:48:23.531235 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:48:23.531214 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634\": container with ID starting with c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634 not found: ID does not exist" containerID="c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634" Sep 15 11:48:23.531311 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.531242 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634"} err="failed to get container status \"c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634\": rpc error: code = NotFound desc = could not find container \"c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634\": container with ID starting with c4107d03ad9f04f6ba1acccdf972569b30f0eb000ba16c9aef45d33c0cf90634 not found: ID does not exist" Sep 15 11:48:23.539166 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.538951 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-br2fj"] Sep 15 11:48:23.540338 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:23.540320 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-br2fj"] Sep 15 11:48:25.187347 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:25.187319 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d85a033b-5293-4480-be91-fab88f9268cc" path="/var/lib/kubelet/pods/d85a033b-5293-4480-be91-fab88f9268cc/volumes" Sep 15 11:48:30.507662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:30.507626 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:31.601560 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:31.601530 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t5tp6"] Sep 15 11:48:31.602102 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:31.601747 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" podUID="6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9" containerName="limitador" containerID="cri-o://8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e" gracePeriod=30 Sep 15 11:48:32.145960 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.145936 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:32.210256 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.210225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-config-file\") pod \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\" (UID: \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\") " Sep 15 11:48:32.210466 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.210309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjvnl\" (UniqueName: \"kubernetes.io/projected/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-kube-api-access-qjvnl\") pod \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\" (UID: \"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9\") " Sep 15 11:48:32.210679 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.210654 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-config-file" (OuterVolumeSpecName: "config-file") pod "6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9" (UID: "6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:48:32.212493 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.212467 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-kube-api-access-qjvnl" (OuterVolumeSpecName: "kube-api-access-qjvnl") pod "6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9" (UID: "6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9"). InnerVolumeSpecName "kube-api-access-qjvnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:48:32.311760 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.311728 2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-config-file\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:48:32.311760 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.311754 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvnl\" (UniqueName: \"kubernetes.io/projected/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9-kube-api-access-qjvnl\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:48:32.566283 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.566168 2579 generic.go:356] "Generic (PLEG): container finished" podID="6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9" containerID="8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e" exitCode=0 Sep 15 11:48:32.566283 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.566243 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" Sep 15 11:48:32.566283 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.566251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" event={"ID":"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9","Type":"ContainerDied","Data":"8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e"} Sep 15 11:48:32.566548 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.566303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-t5tp6" event={"ID":"6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9","Type":"ContainerDied","Data":"be1e443e80609ac338a0f401475b835e5c95721bbe8b7cf8fa2351942fa4436f"} Sep 15 11:48:32.566548 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.566325 2579 scope.go:117] "RemoveContainer" containerID="8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e" Sep 15 11:48:32.574778 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.574760 2579 scope.go:117] "RemoveContainer" containerID="8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e" Sep 15 11:48:32.575034 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:48:32.575016 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e\": container with ID starting with 8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e not found: ID does not exist" containerID="8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e" Sep 15 11:48:32.575102 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.575041 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e"} err="failed to get container status \"8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e\": rpc error: code = NotFound desc = could not find container \"8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e\": container with ID starting with 8bef33084c693f221814b34e3c8a6530b0197e8c6ccf68575e1cc443fe413d7e not found: ID does not exist" Sep 15 11:48:32.592169 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.592139 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t5tp6"] Sep 15 11:48:32.600339 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:32.600313 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-t5tp6"] Sep 15 11:48:33.111851 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.111815 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:48:33.115400 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.115381 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:48:33.187862 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.187834 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9" path="/var/lib/kubelet/pods/6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9/volumes" Sep 15 11:48:33.283871 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.283834 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr"] Sep 15 11:48:33.284244 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.284225 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9" containerName="limitador" Sep 15 11:48:33.284244 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.284243 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9" containerName="limitador" Sep 15 11:48:33.284414 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.284259 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85a033b-5293-4480-be91-fab88f9268cc" containerName="authorino" Sep 15 11:48:33.284414 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.284266 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85a033b-5293-4480-be91-fab88f9268cc" containerName="authorino" Sep 15 11:48:33.284414 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.284346 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afbe7a3-165b-461b-a1d4-8b16f8b2a6c9" containerName="limitador" Sep 15 11:48:33.284414 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.284358 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d85a033b-5293-4480-be91-fab88f9268cc" containerName="authorino" Sep 15 11:48:33.288921 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.288895 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:33.291378 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.291360 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 15 11:48:33.291519 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.291502 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 15 11:48:33.292239 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.292219 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gkn4l\"" Sep 15 11:48:33.292348 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.292230 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 15 11:48:33.297020 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.296998 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr"] Sep 15 11:48:33.420915 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.420827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/febc6a96-0392-44b4-ab90-32c16d2f5a82-data\") pod \"postgres-69fb4594cf-hbbjr\" (UID: \"febc6a96-0392-44b4-ab90-32c16d2f5a82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:33.420915 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.420882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf5m2\" (UniqueName: \"kubernetes.io/projected/febc6a96-0392-44b4-ab90-32c16d2f5a82-kube-api-access-sf5m2\") pod \"postgres-69fb4594cf-hbbjr\" (UID: \"febc6a96-0392-44b4-ab90-32c16d2f5a82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:33.522312 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.522280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/febc6a96-0392-44b4-ab90-32c16d2f5a82-data\") pod \"postgres-69fb4594cf-hbbjr\" (UID: \"febc6a96-0392-44b4-ab90-32c16d2f5a82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:33.522490 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.522333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf5m2\" (UniqueName: \"kubernetes.io/projected/febc6a96-0392-44b4-ab90-32c16d2f5a82-kube-api-access-sf5m2\") pod \"postgres-69fb4594cf-hbbjr\" (UID: \"febc6a96-0392-44b4-ab90-32c16d2f5a82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:33.522699 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.522678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/febc6a96-0392-44b4-ab90-32c16d2f5a82-data\") pod \"postgres-69fb4594cf-hbbjr\" (UID: \"febc6a96-0392-44b4-ab90-32c16d2f5a82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:33.530661 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.530641 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf5m2\" (UniqueName: \"kubernetes.io/projected/febc6a96-0392-44b4-ab90-32c16d2f5a82-kube-api-access-sf5m2\") pod \"postgres-69fb4594cf-hbbjr\" (UID: \"febc6a96-0392-44b4-ab90-32c16d2f5a82\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:33.599313 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.599263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:33.868280 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:33.868249 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr"] Sep 15 11:48:33.871182 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:48:33.871156 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebc6a96_0392_44b4_ab90_32c16d2f5a82.slice/crio-38d9172a1f70c0ba8cdac2921f6bede0af8994470e2c7061b23961245de29c5a WatchSource:0}: Error finding container 38d9172a1f70c0ba8cdac2921f6bede0af8994470e2c7061b23961245de29c5a: Status 404 returned error can't find the container with id 38d9172a1f70c0ba8cdac2921f6bede0af8994470e2c7061b23961245de29c5a Sep 15 11:48:34.577015 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:34.576984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" event={"ID":"febc6a96-0392-44b4-ab90-32c16d2f5a82","Type":"ContainerStarted","Data":"38d9172a1f70c0ba8cdac2921f6bede0af8994470e2c7061b23961245de29c5a"} Sep 15 11:48:36.588545 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:36.588453 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" event={"ID":"febc6a96-0392-44b4-ab90-32c16d2f5a82","Type":"ContainerStarted","Data":"5027e34a29ce25bb5715d0fc44f1c4cbe660adf5f4bc3f10a572091319315378"} Sep 15 11:48:36.588897 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:36.588543 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:36.605337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:36.605288 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" podStartSLOduration=1.1930130349999999 podStartE2EDuration="3.605272963s" totalImagesPullingTime="2.412259928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:36.604456444 +0000 UTC m=+603.949565043" watchObservedRunningTime="2026-09-15 11:48:36.605272963 +0000 UTC m=+603.950381567" Sep 15 11:48:42.621444 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:42.621414 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hbbjr" Sep 15 11:48:43.170827 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.170794 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-ng9nf"] Sep 15 11:48:43.176560 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.176535 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ng9nf" Sep 15 11:48:43.178616 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.178598 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5lr4b\"" Sep 15 11:48:43.188337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.188317 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-ng9nf"] Sep 15 11:48:43.261160 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.261126 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ng9nf"] Sep 15 11:48:43.261404 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:48:43.261385 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-v4f2s], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-ng9nf" podUID="bd0c9dca-904e-416f-bbf7-34db4ca9db1a" Sep 15 11:48:43.289314 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.289284 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-749gl"] Sep 15 11:48:43.293626 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.293606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.295727 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.295709 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 15 11:48:43.301800 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.301778 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-749gl"] Sep 15 11:48:43.309845 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.309824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4f2s\" (UniqueName: \"kubernetes.io/projected/bd0c9dca-904e-416f-bbf7-34db4ca9db1a-kube-api-access-v4f2s\") pod \"authorino-557f876f69-ng9nf\" (UID: \"bd0c9dca-904e-416f-bbf7-34db4ca9db1a\") " pod="kuadrant-system/authorino-557f876f69-ng9nf" Sep 15 11:48:43.393005 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.392974 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-749gl"] Sep 15 11:48:43.393238 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:48:43.393219 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vzrqg tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-749gl" podUID="144bccc9-7c26-4605-92a5-990639865914" Sep 15 11:48:43.410480 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.410450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrqg\" (UniqueName: \"kubernetes.io/projected/144bccc9-7c26-4605-92a5-990639865914-kube-api-access-vzrqg\") pod \"authorino-6d4dfc649f-749gl\" (UID: \"144bccc9-7c26-4605-92a5-990639865914\") " pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.410641 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.410494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4f2s\" (UniqueName: \"kubernetes.io/projected/bd0c9dca-904e-416f-bbf7-34db4ca9db1a-kube-api-access-v4f2s\") pod \"authorino-557f876f69-ng9nf\" (UID: \"bd0c9dca-904e-416f-bbf7-34db4ca9db1a\") " pod="kuadrant-system/authorino-557f876f69-ng9nf" Sep 15 11:48:43.410641 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.410590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/144bccc9-7c26-4605-92a5-990639865914-tls-cert\") pod \"authorino-6d4dfc649f-749gl\" (UID: \"144bccc9-7c26-4605-92a5-990639865914\") " pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.421833 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.421757 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4f2s\" (UniqueName: \"kubernetes.io/projected/bd0c9dca-904e-416f-bbf7-34db4ca9db1a-kube-api-access-v4f2s\") pod \"authorino-557f876f69-ng9nf\" (UID: \"bd0c9dca-904e-416f-bbf7-34db4ca9db1a\") " pod="kuadrant-system/authorino-557f876f69-ng9nf" Sep 15 11:48:43.426471 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.426446 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57cf9d89cb-nqmbx"] Sep 15 11:48:43.430556 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.430539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:48:43.436427 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.436405 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57cf9d89cb-nqmbx"] Sep 15 11:48:43.511627 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.511593 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/144bccc9-7c26-4605-92a5-990639865914-tls-cert\") pod \"authorino-6d4dfc649f-749gl\" (UID: \"144bccc9-7c26-4605-92a5-990639865914\") " pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.511627 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.511631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrqg\" (UniqueName: \"kubernetes.io/projected/144bccc9-7c26-4605-92a5-990639865914-kube-api-access-vzrqg\") pod \"authorino-6d4dfc649f-749gl\" (UID: \"144bccc9-7c26-4605-92a5-990639865914\") " pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.514241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.514195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/144bccc9-7c26-4605-92a5-990639865914-tls-cert\") pod \"authorino-6d4dfc649f-749gl\" (UID: \"144bccc9-7c26-4605-92a5-990639865914\") " pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.519923 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.519901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrqg\" (UniqueName: \"kubernetes.io/projected/144bccc9-7c26-4605-92a5-990639865914-kube-api-access-vzrqg\") pod \"authorino-6d4dfc649f-749gl\" (UID: \"144bccc9-7c26-4605-92a5-990639865914\") " pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.612196 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.612159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-tls-cert\") pod \"authorino-57cf9d89cb-nqmbx\" (UID: \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\") " pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:48:43.612373 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.612261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmqmm\" (UniqueName: \"kubernetes.io/projected/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-kube-api-access-wmqmm\") pod \"authorino-57cf9d89cb-nqmbx\" (UID: \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\") " pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:48:43.619491 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.619467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.619622 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.619468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ng9nf" Sep 15 11:48:43.627260 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.627236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:43.629816 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.629801 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ng9nf" Sep 15 11:48:43.713264 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.713238 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzrqg\" (UniqueName: \"kubernetes.io/projected/144bccc9-7c26-4605-92a5-990639865914-kube-api-access-vzrqg\") pod \"144bccc9-7c26-4605-92a5-990639865914\" (UID: \"144bccc9-7c26-4605-92a5-990639865914\") " Sep 15 11:48:43.713440 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.713301 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/144bccc9-7c26-4605-92a5-990639865914-tls-cert\") pod \"144bccc9-7c26-4605-92a5-990639865914\" (UID: \"144bccc9-7c26-4605-92a5-990639865914\") " Sep 15 11:48:43.713506 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.713464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmqmm\" (UniqueName: \"kubernetes.io/projected/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-kube-api-access-wmqmm\") pod \"authorino-57cf9d89cb-nqmbx\" (UID: \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\") " pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:48:43.713567 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.713520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-tls-cert\") pod \"authorino-57cf9d89cb-nqmbx\" (UID: \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\") " pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:48:43.715438 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.715409 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144bccc9-7c26-4605-92a5-990639865914-kube-api-access-vzrqg" (OuterVolumeSpecName: "kube-api-access-vzrqg") pod "144bccc9-7c26-4605-92a5-990639865914" (UID: "144bccc9-7c26-4605-92a5-990639865914"). InnerVolumeSpecName "kube-api-access-vzrqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:48:43.715438 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.715421 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144bccc9-7c26-4605-92a5-990639865914-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "144bccc9-7c26-4605-92a5-990639865914" (UID: "144bccc9-7c26-4605-92a5-990639865914"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:48:43.715954 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.715935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-tls-cert\") pod \"authorino-57cf9d89cb-nqmbx\" (UID: \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\") " pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:48:43.725100 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.725080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmqmm\" (UniqueName: \"kubernetes.io/projected/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-kube-api-access-wmqmm\") pod \"authorino-57cf9d89cb-nqmbx\" (UID: \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\") " pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:48:43.741098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.741074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:48:43.814611 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.814578 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4f2s\" (UniqueName: \"kubernetes.io/projected/bd0c9dca-904e-416f-bbf7-34db4ca9db1a-kube-api-access-v4f2s\") pod \"bd0c9dca-904e-416f-bbf7-34db4ca9db1a\" (UID: \"bd0c9dca-904e-416f-bbf7-34db4ca9db1a\") " Sep 15 11:48:43.814781 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.814753 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/144bccc9-7c26-4605-92a5-990639865914-tls-cert\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:48:43.814781 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.814765 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrqg\" (UniqueName: \"kubernetes.io/projected/144bccc9-7c26-4605-92a5-990639865914-kube-api-access-vzrqg\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:48:43.816879 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.816857 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0c9dca-904e-416f-bbf7-34db4ca9db1a-kube-api-access-v4f2s" (OuterVolumeSpecName: "kube-api-access-v4f2s") pod "bd0c9dca-904e-416f-bbf7-34db4ca9db1a" (UID: "bd0c9dca-904e-416f-bbf7-34db4ca9db1a"). InnerVolumeSpecName "kube-api-access-v4f2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:48:43.915397 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:43.915363 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4f2s\" (UniqueName: \"kubernetes.io/projected/bd0c9dca-904e-416f-bbf7-34db4ca9db1a-kube-api-access-v4f2s\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:48:44.012079 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.012054 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57cf9d89cb-nqmbx"] Sep 15 11:48:44.014564 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:48:44.014541 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ea198a8_6d9e_495e_a5d3_340d5a6ba45b.slice/crio-d2bf6c6a853d3c7691e8b456e3d80d72471b4c62d241bb354db184bb3d2dfa92 WatchSource:0}: Error finding container d2bf6c6a853d3c7691e8b456e3d80d72471b4c62d241bb354db184bb3d2dfa92: Status 404 returned error can't find the container with id d2bf6c6a853d3c7691e8b456e3d80d72471b4c62d241bb354db184bb3d2dfa92 Sep 15 11:48:44.624778 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.624742 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" event={"ID":"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b","Type":"ContainerStarted","Data":"9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894"} Sep 15 11:48:44.624778 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.624766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-749gl" Sep 15 11:48:44.624778 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.624796 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" event={"ID":"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b","Type":"ContainerStarted","Data":"d2bf6c6a853d3c7691e8b456e3d80d72471b4c62d241bb354db184bb3d2dfa92"} Sep 15 11:48:44.625028 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.624754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ng9nf" Sep 15 11:48:44.641695 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.641647 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" podStartSLOduration=1.230034385 podStartE2EDuration="1.641630667s" totalImagesPullingTime="411.596282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:44.640515586 +0000 UTC m=+611.985624190" watchObservedRunningTime="2026-09-15 11:48:44.641630667 +0000 UTC m=+611.986739272" Sep 15 11:48:44.665435 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.665399 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ng9nf"] Sep 15 11:48:44.675544 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.675514 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-ng9nf"] Sep 15 11:48:44.700674 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.700646 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-749gl"] Sep 15 11:48:44.702789 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:44.702756 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-749gl"] Sep 15 11:48:45.191973 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:45.191943 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144bccc9-7c26-4605-92a5-990639865914" path="/var/lib/kubelet/pods/144bccc9-7c26-4605-92a5-990639865914/volumes" Sep 15 11:48:45.192176 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:45.192163 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd0c9dca-904e-416f-bbf7-34db4ca9db1a" path="/var/lib/kubelet/pods/bd0c9dca-904e-416f-bbf7-34db4ca9db1a/volumes" Sep 15 11:48:49.451039 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.451006 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-744fd7bdd4-x8kfj"] Sep 15 11:48:49.459607 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.459584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.461674 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.461650 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 15 11:48:49.461818 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.461722 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 15 11:48:49.461818 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.461748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rnnhv\"" Sep 15 11:48:49.461818 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.461780 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 15 11:48:49.465116 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.465093 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-744fd7bdd4-x8kfj"] Sep 15 11:48:49.560528 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.560499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.560707 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.560534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2vz\" (UniqueName: \"kubernetes.io/projected/dd509ba2-9bb0-4210-8935-0010c872cf12-kube-api-access-fj2vz\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.560707 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.560554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-metrics-certs\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.613446 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.613414 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-777d95b559-qkjvn"] Sep 15 11:48:49.616647 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.616633 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.626621 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.626599 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777d95b559-qkjvn"] Sep 15 11:48:49.661376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.661344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.661535 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.661388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2vz\" (UniqueName: \"kubernetes.io/projected/dd509ba2-9bb0-4210-8935-0010c872cf12-kube-api-access-fj2vz\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.661535 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.661404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-metrics-certs\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.661535 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:48:49.661480 2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 15 11:48:49.661535 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:48:49.661536 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert podName:dd509ba2-9bb0-4210-8935-0010c872cf12 nodeName:}" failed. No retries permitted until 2026-09-15 11:48:50.161519589 +0000 UTC m=+617.506628172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert") pod "maas-controller-744fd7bdd4-x8kfj" (UID: "dd509ba2-9bb0-4210-8935-0010c872cf12") : secret "maas-controller-webhook-cert" not found Sep 15 11:48:49.663851 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.663833 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-metrics-certs\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.669793 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.669773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2vz\" (UniqueName: \"kubernetes.io/projected/dd509ba2-9bb0-4210-8935-0010c872cf12-kube-api-access-fj2vz\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:49.762083 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.762009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-metrics-certs\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.762083 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.762051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-webhook-cert\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.762267 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.762137 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnkn6\" (UniqueName: \"kubernetes.io/projected/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-kube-api-access-jnkn6\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.863596 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.863560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-metrics-certs\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.863777 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.863607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-webhook-cert\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.863777 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.863665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnkn6\" (UniqueName: \"kubernetes.io/projected/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-kube-api-access-jnkn6\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.866218 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.866181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-metrics-certs\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.866368 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.866348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-webhook-cert\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.872630 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.872608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnkn6\" (UniqueName: \"kubernetes.io/projected/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-kube-api-access-jnkn6\") pod \"maas-controller-777d95b559-qkjvn\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:49.927404 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:49.927376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:50.166323 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:50.166235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:50.168730 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:50.168713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert\") pod \"maas-controller-744fd7bdd4-x8kfj\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:50.196650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:50.196628 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777d95b559-qkjvn"] Sep 15 11:48:50.198781 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:48:50.198745 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf8de17_cd85_4b51_a039_eeed6ed8c3ba.slice/crio-f73dc71f46c9f525f373df719898d2cbf5315d7d3a6c931c64b3204bbcffb802 WatchSource:0}: Error finding container f73dc71f46c9f525f373df719898d2cbf5315d7d3a6c931c64b3204bbcffb802: Status 404 returned error can't find the container with id f73dc71f46c9f525f373df719898d2cbf5315d7d3a6c931c64b3204bbcffb802 Sep 15 11:48:50.370732 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:50.370703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:50.647058 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:50.646950 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-744fd7bdd4-x8kfj"] Sep 15 11:48:50.654595 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:48:50.654470 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd509ba2_9bb0_4210_8935_0010c872cf12.slice/crio-75ee1ad7f2aff4b8d6747759d60c19a0ebc4188d6e43b55105c47d63bcb49134 WatchSource:0}: Error finding container 75ee1ad7f2aff4b8d6747759d60c19a0ebc4188d6e43b55105c47d63bcb49134: Status 404 returned error can't find the container with id 75ee1ad7f2aff4b8d6747759d60c19a0ebc4188d6e43b55105c47d63bcb49134 Sep 15 11:48:50.656622 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:50.656596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-qkjvn" event={"ID":"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba","Type":"ContainerStarted","Data":"f73dc71f46c9f525f373df719898d2cbf5315d7d3a6c931c64b3204bbcffb802"} Sep 15 11:48:51.664892 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:51.664834 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" event={"ID":"dd509ba2-9bb0-4210-8935-0010c872cf12","Type":"ContainerStarted","Data":"75ee1ad7f2aff4b8d6747759d60c19a0ebc4188d6e43b55105c47d63bcb49134"} Sep 15 11:48:53.675561 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:53.675525 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-qkjvn" event={"ID":"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba","Type":"ContainerStarted","Data":"a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f"} Sep 15 11:48:53.676030 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:53.675614 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:48:53.677026 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:53.677006 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" event={"ID":"dd509ba2-9bb0-4210-8935-0010c872cf12","Type":"ContainerStarted","Data":"1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309"} Sep 15 11:48:53.677101 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:53.677062 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:48:53.697271 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:53.697196 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-777d95b559-qkjvn" podStartSLOduration=1.9968747690000002 podStartE2EDuration="4.697180907s" totalImagesPullingTime="2.700306138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:53.695970508 +0000 UTC m=+621.041079109" watchObservedRunningTime="2026-09-15 11:48:53.697180907 +0000 UTC m=+621.042289539" Sep 15 11:48:53.711939 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:48:53.711895 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" podStartSLOduration=2.456649067 podStartE2EDuration="4.711879082s" totalImagesPullingTime="2.255230015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:53.711471538 +0000 UTC m=+621.056580140" watchObservedRunningTime="2026-09-15 11:48:53.711879082 +0000 UTC m=+621.056987734" Sep 15 11:49:04.683312 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:04.683232 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:49:04.683312 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:04.683302 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 11:49:04.754739 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:04.754700 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-744fd7bdd4-x8kfj"] Sep 15 11:49:04.754922 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:04.754898 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" podUID="dd509ba2-9bb0-4210-8935-0010c872cf12" containerName="manager" containerID="cri-o://1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309" gracePeriod=30 Sep 15 11:49:04.987578 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:04.987556 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:49:05.091298 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.091269 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj2vz\" (UniqueName: \"kubernetes.io/projected/dd509ba2-9bb0-4210-8935-0010c872cf12-kube-api-access-fj2vz\") pod \"dd509ba2-9bb0-4210-8935-0010c872cf12\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " Sep 15 11:49:05.091492 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.091312 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-metrics-certs\") pod \"dd509ba2-9bb0-4210-8935-0010c872cf12\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " Sep 15 11:49:05.091492 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.091382 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert\") pod \"dd509ba2-9bb0-4210-8935-0010c872cf12\" (UID: \"dd509ba2-9bb0-4210-8935-0010c872cf12\") " Sep 15 11:49:05.093653 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.093618 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dd509ba2-9bb0-4210-8935-0010c872cf12" (UID: "dd509ba2-9bb0-4210-8935-0010c872cf12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:49:05.093759 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.093656 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dd509ba2-9bb0-4210-8935-0010c872cf12" (UID: "dd509ba2-9bb0-4210-8935-0010c872cf12"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:49:05.093759 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.093661 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd509ba2-9bb0-4210-8935-0010c872cf12-kube-api-access-fj2vz" (OuterVolumeSpecName: "kube-api-access-fj2vz") pod "dd509ba2-9bb0-4210-8935-0010c872cf12" (UID: "dd509ba2-9bb0-4210-8935-0010c872cf12"). InnerVolumeSpecName "kube-api-access-fj2vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:49:05.192220 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.192173 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-webhook-cert\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:49:05.192220 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.192224 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj2vz\" (UniqueName: \"kubernetes.io/projected/dd509ba2-9bb0-4210-8935-0010c872cf12-kube-api-access-fj2vz\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:49:05.192417 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.192240 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd509ba2-9bb0-4210-8935-0010c872cf12-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:49:05.734216 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.734165 2579 generic.go:356] "Generic (PLEG): container finished" podID="dd509ba2-9bb0-4210-8935-0010c872cf12" containerID="1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309" exitCode=0 Sep 15 11:49:05.734600 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.734242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" event={"ID":"dd509ba2-9bb0-4210-8935-0010c872cf12","Type":"ContainerDied","Data":"1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309"} Sep 15 11:49:05.734600 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.734286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" event={"ID":"dd509ba2-9bb0-4210-8935-0010c872cf12","Type":"ContainerDied","Data":"75ee1ad7f2aff4b8d6747759d60c19a0ebc4188d6e43b55105c47d63bcb49134"} Sep 15 11:49:05.734600 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.734307 2579 scope.go:117] "RemoveContainer" containerID="1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309" Sep 15 11:49:05.734600 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.734247 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-744fd7bdd4-x8kfj" Sep 15 11:49:05.742396 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.742369 2579 scope.go:117] "RemoveContainer" containerID="1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309" Sep 15 11:49:05.742683 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:49:05.742665 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309\": container with ID starting with 1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309 not found: ID does not exist" containerID="1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309" Sep 15 11:49:05.742736 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.742692 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309"} err="failed to get container status \"1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309\": rpc error: code = NotFound desc = could not find container \"1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309\": container with ID starting with 1577a175f60d1de9ebaec7479e6c92898333eee2c3bc259eb322240d9c7c0309 not found: ID does not exist" Sep 15 11:49:05.752636 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.752580 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-744fd7bdd4-x8kfj"] Sep 15 11:49:05.754510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:05.754487 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-744fd7bdd4-x8kfj"] Sep 15 11:49:07.188455 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:07.188423 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd509ba2-9bb0-4210-8935-0010c872cf12" path="/var/lib/kubelet/pods/dd509ba2-9bb0-4210-8935-0010c872cf12/volumes" Sep 15 11:49:27.326106 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.326070 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg"] Sep 15 11:49:27.328364 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.326399 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd509ba2-9bb0-4210-8935-0010c872cf12" containerName="manager" Sep 15 11:49:27.328364 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.326411 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd509ba2-9bb0-4210-8935-0010c872cf12" containerName="manager" Sep 15 11:49:27.328364 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.326472 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd509ba2-9bb0-4210-8935-0010c872cf12" containerName="manager" Sep 15 11:49:27.329141 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.329127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.331293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.331274 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 15 11:49:27.332213 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.332182 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2npzq\"" Sep 15 11:49:27.332331 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.332314 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 15 11:49:27.337881 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.337863 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg"] Sep 15 11:49:27.481122 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.481090 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-metrics-certs\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.481324 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.481164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-maas-api-tls\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.481324 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.481189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2lp\" (UniqueName: \"kubernetes.io/projected/ebd13dcd-d3a3-42eb-807b-6d43727411d4-kube-api-access-sl2lp\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.481324 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.481241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd13dcd-d3a3-42eb-807b-6d43727411d4-openshift-service-ca\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.581824 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.581750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-maas-api-tls\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.581824 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.581791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2lp\" (UniqueName: \"kubernetes.io/projected/ebd13dcd-d3a3-42eb-807b-6d43727411d4-kube-api-access-sl2lp\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.582035 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.581840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd13dcd-d3a3-42eb-807b-6d43727411d4-openshift-service-ca\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.582035 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.581858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-metrics-certs\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.582611 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.582530 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd13dcd-d3a3-42eb-807b-6d43727411d4-openshift-service-ca\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.584434 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.584415 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-maas-api-tls\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.584549 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.584530 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-metrics-certs\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.590925 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.590901 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2lp\" (UniqueName: \"kubernetes.io/projected/ebd13dcd-d3a3-42eb-807b-6d43727411d4-kube-api-access-sl2lp\") pod \"maas-api-6c49d8454d-xsmjg\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.639133 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.639104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:27.909035 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.908960 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj"] Sep 15 11:49:27.912358 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.912341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:27.916495 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.916474 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 15 11:49:27.919789 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.919770 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-78n4z\"" Sep 15 11:49:27.946241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.946213 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj"] Sep 15 11:49:27.948581 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.948472 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg"] Sep 15 11:49:27.951144 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:49:27.951118 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd13dcd_d3a3_42eb_807b_6d43727411d4.slice/crio-2b4ad0966e05c7a2a61e8697793b70bf23fd5a556ed324154f9d17020f806230 WatchSource:0}: Error finding container 2b4ad0966e05c7a2a61e8697793b70bf23fd5a556ed324154f9d17020f806230: Status 404 returned error can't find the container with id 2b4ad0966e05c7a2a61e8697793b70bf23fd5a556ed324154f9d17020f806230 Sep 15 11:49:27.952992 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:27.952972 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 11:49:28.086058 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.086028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4wk\" (UniqueName: \"kubernetes.io/projected/c6f5fe03-607a-4f4a-9a59-f176bae5c3cc-kube-api-access-dn4wk\") pod \"payload-pre-processing-59fbc47bf8-j2rxj\" (UID: \"c6f5fe03-607a-4f4a-9a59-f176bae5c3cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:28.086254 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.086091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f5fe03-607a-4f4a-9a59-f176bae5c3cc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-j2rxj\" (UID: \"c6f5fe03-607a-4f4a-9a59-f176bae5c3cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:28.187503 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.187464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f5fe03-607a-4f4a-9a59-f176bae5c3cc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-j2rxj\" (UID: \"c6f5fe03-607a-4f4a-9a59-f176bae5c3cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:28.187700 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.187563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4wk\" (UniqueName: \"kubernetes.io/projected/c6f5fe03-607a-4f4a-9a59-f176bae5c3cc-kube-api-access-dn4wk\") pod \"payload-pre-processing-59fbc47bf8-j2rxj\" (UID: \"c6f5fe03-607a-4f4a-9a59-f176bae5c3cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:28.188279 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.188256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6f5fe03-607a-4f4a-9a59-f176bae5c3cc-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-j2rxj\" (UID: \"c6f5fe03-607a-4f4a-9a59-f176bae5c3cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:28.197461 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.197434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4wk\" (UniqueName: \"kubernetes.io/projected/c6f5fe03-607a-4f4a-9a59-f176bae5c3cc-kube-api-access-dn4wk\") pod \"payload-pre-processing-59fbc47bf8-j2rxj\" (UID: \"c6f5fe03-607a-4f4a-9a59-f176bae5c3cc\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:28.223251 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.223217 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:28.495942 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.495869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj"] Sep 15 11:49:28.499296 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:49:28.499269 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f5fe03_607a_4f4a_9a59_f176bae5c3cc.slice/crio-227f81b2efe8e398c0c2dbc7ec2452ce9c8ac263cd88d932d59ad163e6f3aa57 WatchSource:0}: Error finding container 227f81b2efe8e398c0c2dbc7ec2452ce9c8ac263cd88d932d59ad163e6f3aa57: Status 404 returned error can't find the container with id 227f81b2efe8e398c0c2dbc7ec2452ce9c8ac263cd88d932d59ad163e6f3aa57 Sep 15 11:49:28.836541 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.836455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" event={"ID":"c6f5fe03-607a-4f4a-9a59-f176bae5c3cc","Type":"ContainerStarted","Data":"227f81b2efe8e398c0c2dbc7ec2452ce9c8ac263cd88d932d59ad163e6f3aa57"} Sep 15 11:49:28.837979 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:28.837954 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" event={"ID":"ebd13dcd-d3a3-42eb-807b-6d43727411d4","Type":"ContainerStarted","Data":"2b4ad0966e05c7a2a61e8697793b70bf23fd5a556ed324154f9d17020f806230"} Sep 15 11:49:30.852067 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:30.852027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" event={"ID":"c6f5fe03-607a-4f4a-9a59-f176bae5c3cc","Type":"ContainerStarted","Data":"a94c337ad97da9b88f934632afa44d11f3b9509a5bc6cf49ee4a5d47d6b318df"} Sep 15 11:49:30.852548 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:30.852147 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:30.853662 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:30.853640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" event={"ID":"ebd13dcd-d3a3-42eb-807b-6d43727411d4","Type":"ContainerStarted","Data":"ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f"} Sep 15 11:49:30.853779 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:30.853717 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:30.870028 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:30.869983 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" podStartSLOduration=2.59537973 podStartE2EDuration="3.869968991s" totalImagesPullingTime="1.274589261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:49:30.868439953 +0000 UTC m=+658.213548562" watchObservedRunningTime="2026-09-15 11:49:30.869968991 +0000 UTC m=+658.215077595" Sep 15 11:49:30.885485 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:30.885439 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" podStartSLOduration=2.064928912 podStartE2EDuration="3.885424207s" totalImagesPullingTime="1.820495295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:49:30.884325875 +0000 UTC m=+658.229434480" watchObservedRunningTime="2026-09-15 11:49:30.885424207 +0000 UTC m=+658.230532811" Sep 15 11:49:36.863258 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:36.863231 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 11:49:41.859940 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:41.859912 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-j2rxj" Sep 15 11:49:43.469555 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.469525 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6"] Sep 15 11:49:43.474108 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.474093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.476689 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.476669 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 15 11:49:43.477600 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.477584 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wkm2k\"" Sep 15 11:49:43.477650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.477624 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 15 11:49:43.477650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.477639 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 15 11:49:43.481184 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.481127 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6"] Sep 15 11:49:43.514238 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.514186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.514382 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.514244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.514382 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.514277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.514382 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.514322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7300a517-6504-4680-a82e-f847ae0283a4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.514382 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.514356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trntg\" (UniqueName: \"kubernetes.io/projected/7300a517-6504-4680-a82e-f847ae0283a4-kube-api-access-trntg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.514382 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.514372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.614833 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.614794 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.614833 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.614836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.615023 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.614860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.615023 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.614932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7300a517-6504-4680-a82e-f847ae0283a4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.615023 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.614986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trntg\" (UniqueName: \"kubernetes.io/projected/7300a517-6504-4680-a82e-f847ae0283a4-kube-api-access-trntg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.615023 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.615008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.615321 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.615298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.615420 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.615329 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.615420 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.615351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.617843 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.617810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7300a517-6504-4680-a82e-f847ae0283a4-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.618255 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.618234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7300a517-6504-4680-a82e-f847ae0283a4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.624956 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.624934 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trntg\" (UniqueName: \"kubernetes.io/projected/7300a517-6504-4680-a82e-f847ae0283a4-kube-api-access-trntg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6\" (UID: \"7300a517-6504-4680-a82e-f847ae0283a4\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:43.784577 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:43.784488 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:44.058524 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:44.058443 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6"] Sep 15 11:49:44.061630 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:49:44.061605 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7300a517_6504_4680_a82e_f847ae0283a4.slice/crio-2737846afd87f5ef7d10d53c892b64736e6dd3bf296822ba562b615a29c8047e WatchSource:0}: Error finding container 2737846afd87f5ef7d10d53c892b64736e6dd3bf296822ba562b615a29c8047e: Status 404 returned error can't find the container with id 2737846afd87f5ef7d10d53c892b64736e6dd3bf296822ba562b615a29c8047e Sep 15 11:49:44.921928 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:44.921892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" event={"ID":"7300a517-6504-4680-a82e-f847ae0283a4","Type":"ContainerStarted","Data":"2737846afd87f5ef7d10d53c892b64736e6dd3bf296822ba562b615a29c8047e"} Sep 15 11:49:45.927665 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:45.927632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" event={"ID":"7300a517-6504-4680-a82e-f847ae0283a4","Type":"ContainerStarted","Data":"eaf9fedd980fa4157132b5180fa387337626bee61c9d247683685e90de4a86fb"} Sep 15 11:49:45.945931 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:45.945886 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" podStartSLOduration=1.918376402 podStartE2EDuration="2.945870039s" totalImagesPullingTime="1.027493637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:49:45.944056144 +0000 UTC m=+673.289164750" watchObservedRunningTime="2026-09-15 11:49:45.945870039 +0000 UTC m=+673.290978644" Sep 15 11:49:53.785371 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:53.785330 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:53.785756 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:53.785402 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:53.797515 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:53.797491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:49:53.974887 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:49:53.974858 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6" Sep 15 11:52:27.395885 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.395848 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cb98cc9fc-hsx8x"] Sep 15 11:52:27.398054 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.398034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" Sep 15 11:52:27.413050 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.413022 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cb98cc9fc-hsx8x"] Sep 15 11:52:27.588575 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.588531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc3fbecd-5eeb-4855-9b26-b218c74137ea-tls-cert\") pod \"authorino-cb98cc9fc-hsx8x\" (UID: \"dc3fbecd-5eeb-4855-9b26-b218c74137ea\") " pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" Sep 15 11:52:27.588759 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.588608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lp6n\" (UniqueName: \"kubernetes.io/projected/dc3fbecd-5eeb-4855-9b26-b218c74137ea-kube-api-access-6lp6n\") pod \"authorino-cb98cc9fc-hsx8x\" (UID: \"dc3fbecd-5eeb-4855-9b26-b218c74137ea\") " pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" Sep 15 11:52:27.689499 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.689473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lp6n\" (UniqueName: \"kubernetes.io/projected/dc3fbecd-5eeb-4855-9b26-b218c74137ea-kube-api-access-6lp6n\") pod \"authorino-cb98cc9fc-hsx8x\" (UID: \"dc3fbecd-5eeb-4855-9b26-b218c74137ea\") " pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" Sep 15 11:52:27.689647 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.689540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc3fbecd-5eeb-4855-9b26-b218c74137ea-tls-cert\") pod \"authorino-cb98cc9fc-hsx8x\" (UID: \"dc3fbecd-5eeb-4855-9b26-b218c74137ea\") " pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" Sep 15 11:52:27.692006 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.691966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc3fbecd-5eeb-4855-9b26-b218c74137ea-tls-cert\") pod \"authorino-cb98cc9fc-hsx8x\" (UID: \"dc3fbecd-5eeb-4855-9b26-b218c74137ea\") " pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" Sep 15 11:52:27.700245 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.700189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lp6n\" (UniqueName: \"kubernetes.io/projected/dc3fbecd-5eeb-4855-9b26-b218c74137ea-kube-api-access-6lp6n\") pod \"authorino-cb98cc9fc-hsx8x\" (UID: \"dc3fbecd-5eeb-4855-9b26-b218c74137ea\") " pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" Sep 15 11:52:27.710466 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.710445 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" Sep 15 11:52:27.984343 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:27.984267 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cb98cc9fc-hsx8x"] Sep 15 11:52:27.987585 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:27.987560 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc3fbecd_5eeb_4855_9b26_b218c74137ea.slice/crio-75982cf4a6b7df006fccd2ed381d8e90ddd891675684723b93df69cd288709ac WatchSource:0}: Error finding container 75982cf4a6b7df006fccd2ed381d8e90ddd891675684723b93df69cd288709ac: Status 404 returned error can't find the container with id 75982cf4a6b7df006fccd2ed381d8e90ddd891675684723b93df69cd288709ac Sep 15 11:52:28.658882 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:28.658848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" event={"ID":"dc3fbecd-5eeb-4855-9b26-b218c74137ea","Type":"ContainerStarted","Data":"826b7f0a957cd6819d5a008a3f38258af249982f217cc8ec68db6022df8acb99"} Sep 15 11:52:28.658882 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:28.658887 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" event={"ID":"dc3fbecd-5eeb-4855-9b26-b218c74137ea","Type":"ContainerStarted","Data":"75982cf4a6b7df006fccd2ed381d8e90ddd891675684723b93df69cd288709ac"} Sep 15 11:52:28.676784 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:28.676731 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cb98cc9fc-hsx8x" podStartSLOduration=1.107837225 podStartE2EDuration="1.676714149s" totalImagesPullingTime="568.876924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:28.675604611 +0000 UTC m=+836.020713239" watchObservedRunningTime="2026-09-15 11:52:28.676714149 +0000 UTC m=+836.021822785" Sep 15 11:52:28.715444 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:28.715408 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57cf9d89cb-nqmbx"] Sep 15 11:52:28.715650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:28.715623 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" podUID="6ea198a8-6d9e-495e-a5d3-340d5a6ba45b" containerName="authorino" containerID="cri-o://9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894" gracePeriod=30 Sep 15 11:52:28.977385 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:28.977359 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:52:29.102000 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.101918 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmqmm\" (UniqueName: \"kubernetes.io/projected/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-kube-api-access-wmqmm\") pod \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\" (UID: \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\") " Sep 15 11:52:29.102000 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.101967 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-tls-cert\") pod \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\" (UID: \"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b\") " Sep 15 11:52:29.104792 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.104763 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-kube-api-access-wmqmm" (OuterVolumeSpecName: "kube-api-access-wmqmm") pod "6ea198a8-6d9e-495e-a5d3-340d5a6ba45b" (UID: "6ea198a8-6d9e-495e-a5d3-340d5a6ba45b"). InnerVolumeSpecName "kube-api-access-wmqmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:52:29.113896 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.113853 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6ea198a8-6d9e-495e-a5d3-340d5a6ba45b" (UID: "6ea198a8-6d9e-495e-a5d3-340d5a6ba45b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:52:29.203176 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.203143 2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-tls-cert\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:52:29.203176 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.203167 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmqmm\" (UniqueName: \"kubernetes.io/projected/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b-kube-api-access-wmqmm\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:52:29.665802 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.665770 2579 generic.go:356] "Generic (PLEG): container finished" podID="6ea198a8-6d9e-495e-a5d3-340d5a6ba45b" containerID="9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894" exitCode=0 Sep 15 11:52:29.666263 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.665809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" event={"ID":"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b","Type":"ContainerDied","Data":"9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894"} Sep 15 11:52:29.666263 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.665865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" event={"ID":"6ea198a8-6d9e-495e-a5d3-340d5a6ba45b","Type":"ContainerDied","Data":"d2bf6c6a853d3c7691e8b456e3d80d72471b4c62d241bb354db184bb3d2dfa92"} Sep 15 11:52:29.666263 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.665884 2579 scope.go:117] "RemoveContainer" containerID="9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894" Sep 15 11:52:29.666263 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.665824 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57cf9d89cb-nqmbx" Sep 15 11:52:29.674624 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.674602 2579 scope.go:117] "RemoveContainer" containerID="9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894" Sep 15 11:52:29.674943 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:29.674921 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894\": container with ID starting with 9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894 not found: ID does not exist" containerID="9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894" Sep 15 11:52:29.675024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.674954 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894"} err="failed to get container status \"9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894\": rpc error: code = NotFound desc = could not find container \"9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894\": container with ID starting with 9ce12343b4ec1ebb872c25876d7dcd77e5da524712b2c385fb910bae58100894 not found: ID does not exist" Sep 15 11:52:29.701246 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.701220 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57cf9d89cb-nqmbx"] Sep 15 11:52:29.707935 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:29.707902 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57cf9d89cb-nqmbx"] Sep 15 11:52:31.188007 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:31.187970 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea198a8-6d9e-495e-a5d3-340d5a6ba45b" path="/var/lib/kubelet/pods/6ea198a8-6d9e-495e-a5d3-340d5a6ba45b/volumes" Sep 15 11:52:50.677098 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.677048 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7890-c59673"] Sep 15 11:52:50.677544 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.677443 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea198a8-6d9e-495e-a5d3-340d5a6ba45b" containerName="authorino" Sep 15 11:52:50.677544 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.677456 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea198a8-6d9e-495e-a5d3-340d5a6ba45b" containerName="authorino" Sep 15 11:52:50.677544 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.677514 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea198a8-6d9e-495e-a5d3-340d5a6ba45b" containerName="authorino" Sep 15 11:52:50.680241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.680224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7890-c59673" Sep 15 11:52:50.686122 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.686093 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6twr\"" Sep 15 11:52:50.691475 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.691451 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7890-c59673"] Sep 15 11:52:50.788489 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.788450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7cwj\" (UniqueName: \"kubernetes.io/projected/fa01de0b-63df-422c-b8f0-705b338bcf26-kube-api-access-b7cwj\") pod \"test-curl-7890-c59673\" (UID: \"fa01de0b-63df-422c-b8f0-705b338bcf26\") " pod="opendatahub/test-curl-7890-c59673" Sep 15 11:52:50.889527 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.889476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7cwj\" (UniqueName: \"kubernetes.io/projected/fa01de0b-63df-422c-b8f0-705b338bcf26-kube-api-access-b7cwj\") pod \"test-curl-7890-c59673\" (UID: \"fa01de0b-63df-422c-b8f0-705b338bcf26\") " pod="opendatahub/test-curl-7890-c59673" Sep 15 11:52:50.907128 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.907097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7cwj\" (UniqueName: \"kubernetes.io/projected/fa01de0b-63df-422c-b8f0-705b338bcf26-kube-api-access-b7cwj\") pod \"test-curl-7890-c59673\" (UID: \"fa01de0b-63df-422c-b8f0-705b338bcf26\") " pod="opendatahub/test-curl-7890-c59673" Sep 15 11:52:50.988989 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:50.988909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7890-c59673" Sep 15 11:52:51.269731 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:51.269691 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7890-c59673"] Sep 15 11:52:51.272149 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:51.272121 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa01de0b_63df_422c_b8f0_705b338bcf26.slice/crio-c95aa978e9d5a7e9ff8c27ef56301736abfc6e326fa9af40e33ded587e68f13e WatchSource:0}: Error finding container c95aa978e9d5a7e9ff8c27ef56301736abfc6e326fa9af40e33ded587e68f13e: Status 404 returned error can't find the container with id c95aa978e9d5a7e9ff8c27ef56301736abfc6e326fa9af40e33ded587e68f13e Sep 15 11:52:51.770008 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:51.769975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7890-c59673" event={"ID":"fa01de0b-63df-422c-b8f0-705b338bcf26","Type":"ContainerStarted","Data":"c95aa978e9d5a7e9ff8c27ef56301736abfc6e326fa9af40e33ded587e68f13e"} Sep 15 11:52:53.781134 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:53.781099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7890-c59673" event={"ID":"fa01de0b-63df-422c-b8f0-705b338bcf26","Type":"ContainerStarted","Data":"e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30"} Sep 15 11:52:53.816242 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:53.816161 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7890-c59673" podStartSLOduration=1.875579321 podStartE2EDuration="3.816142664s" totalImagesPullingTime="1.940563343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:53.8125077 +0000 UTC m=+861.157616305" watchObservedRunningTime="2026-09-15 11:52:53.816142664 +0000 UTC m=+861.161251294" Sep 15 11:52:54.245231 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.245165 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq"] Sep 15 11:52:54.247652 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.247629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.250548 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.250518 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-ac76e5-serving-cert\"" Sep 15 11:52:54.272789 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.272756 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq"] Sep 15 11:52:54.323627 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.323524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr2dg\" (UniqueName: \"kubernetes.io/projected/c20014d5-1a23-4a5d-b022-5a0b34192400-kube-api-access-mr2dg\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.323627 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.323568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c20014d5-1a23-4a5d-b022-5a0b34192400-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.323840 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.323659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.323840 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.323704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-metrics-certs\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.425080 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.425046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-metrics-certs\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.425337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.425088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr2dg\" (UniqueName: \"kubernetes.io/projected/c20014d5-1a23-4a5d-b022-5a0b34192400-kube-api-access-mr2dg\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.425337 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.425261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c20014d5-1a23-4a5d-b022-5a0b34192400-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.425488 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.425352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.425488 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:54.425477 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-serving-cert: secret "maas-api-e2e-worker-w4-ac76e5-serving-cert" not found Sep 15 11:52:54.425589 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:54.425541 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls podName:c20014d5-1a23-4a5d-b022-5a0b34192400 nodeName:}" failed. No retries permitted until 2026-09-15 11:52:54.925524222 +0000 UTC m=+862.270632805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls") pod "maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" (UID: "c20014d5-1a23-4a5d-b022-5a0b34192400") : secret "maas-api-e2e-worker-w4-ac76e5-serving-cert" not found Sep 15 11:52:54.425870 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.425852 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c20014d5-1a23-4a5d-b022-5a0b34192400-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.427818 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.427795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-metrics-certs\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.441936 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.441897 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr2dg\" (UniqueName: \"kubernetes.io/projected/c20014d5-1a23-4a5d-b022-5a0b34192400-kube-api-access-mr2dg\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.726353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.726319 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7890-c59673"] Sep 15 11:52:54.732795 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.732768 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7890-c59673"] Sep 15 11:52:54.785937 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:54.785907 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7890-c59673" container="test-curl-7890-c59673" Sep 15 11:52:54.867522 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.867490 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7890-a12d54"] Sep 15 11:52:54.867853 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.867842 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa01de0b-63df-422c-b8f0-705b338bcf26" containerName="test-curl-7890-c59673" Sep 15 11:52:54.867891 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.867855 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa01de0b-63df-422c-b8f0-705b338bcf26" containerName="test-curl-7890-c59673" Sep 15 11:52:54.867928 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.867909 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa01de0b-63df-422c-b8f0-705b338bcf26" containerName="test-curl-7890-c59673" Sep 15 11:52:54.869681 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.869667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7890-a12d54" Sep 15 11:52:54.886721 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.886693 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7890-a12d54"] Sep 15 11:52:54.930228 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.930162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lpmp\" (UniqueName: \"kubernetes.io/projected/b3325f8e-21d6-4519-b994-6464410b83c3-kube-api-access-2lpmp\") pod \"test-curl-7890-a12d54\" (UID: \"b3325f8e-21d6-4519-b994-6464410b83c3\") " pod="opendatahub/test-curl-7890-a12d54" Sep 15 11:52:54.930399 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.930279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:54.932870 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:54.932850 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls\") pod \"maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:55.031111 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.031029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lpmp\" (UniqueName: \"kubernetes.io/projected/b3325f8e-21d6-4519-b994-6464410b83c3-kube-api-access-2lpmp\") pod \"test-curl-7890-a12d54\" (UID: \"b3325f8e-21d6-4519-b994-6464410b83c3\") " pod="opendatahub/test-curl-7890-a12d54" Sep 15 11:52:55.051908 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.051876 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lpmp\" (UniqueName: \"kubernetes.io/projected/b3325f8e-21d6-4519-b994-6464410b83c3-kube-api-access-2lpmp\") pod \"test-curl-7890-a12d54\" (UID: \"b3325f8e-21d6-4519-b994-6464410b83c3\") " pod="opendatahub/test-curl-7890-a12d54" Sep 15 11:52:55.157802 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.157770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:55.186139 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.186107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7890-a12d54" Sep 15 11:52:55.333442 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.333354 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm"] Sep 15 11:52:55.336070 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.336046 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:55.348792 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.348767 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ac76e5\"" Sep 15 11:52:55.348926 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.348912 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ac76e5-dockercfg-tpjrk\"" Sep 15 11:52:55.413945 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.413911 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44"] Sep 15 11:52:55.416166 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.416148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:55.434528 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.434492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tt2\" (UniqueName: \"kubernetes.io/projected/6b41998d-4972-420b-a6de-79f0807b46e6-kube-api-access-t7tt2\") pod \"payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm\" (UID: \"6b41998d-4972-420b-a6de-79f0807b46e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:55.434672 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.434624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b41998d-4972-420b-a6de-79f0807b46e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm\" (UID: \"6b41998d-4972-420b-a6de-79f0807b46e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:55.472949 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.472917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm"] Sep 15 11:52:55.508806 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.508783 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq"] Sep 15 11:52:55.510993 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:55.510966 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20014d5_1a23_4a5d_b022_5a0b34192400.slice/crio-61b267ad8d7540593d8ee3e5e675773023c3316d101fc671c9bb548901797d07 WatchSource:0}: Error finding container 61b267ad8d7540593d8ee3e5e675773023c3316d101fc671c9bb548901797d07: Status 404 returned error can't find the container with id 61b267ad8d7540593d8ee3e5e675773023c3316d101fc671c9bb548901797d07 Sep 15 11:52:55.526353 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:55.526318 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3325f8e_21d6_4519_b994_6464410b83c3.slice/crio-cee9812185ea0644b562ae1a12b773200c5f63288e6ab1808b212734fb8c7317 WatchSource:0}: Error finding container cee9812185ea0644b562ae1a12b773200c5f63288e6ab1808b212734fb8c7317: Status 404 returned error can't find the container with id cee9812185ea0644b562ae1a12b773200c5f63288e6ab1808b212734fb8c7317 Sep 15 11:52:55.532516 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.532487 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7890-a12d54"] Sep 15 11:52:55.536012 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.535987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lbf\" (UniqueName: \"kubernetes.io/projected/1dec157b-20dc-49a1-ba71-f50cb28126b0-kube-api-access-58lbf\") pod \"payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44\" (UID: \"1dec157b-20dc-49a1-ba71-f50cb28126b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:55.536150 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.536033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b41998d-4972-420b-a6de-79f0807b46e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm\" (UID: \"6b41998d-4972-420b-a6de-79f0807b46e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:55.536250 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.536158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tt2\" (UniqueName: \"kubernetes.io/projected/6b41998d-4972-420b-a6de-79f0807b46e6-kube-api-access-t7tt2\") pod \"payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm\" (UID: \"6b41998d-4972-420b-a6de-79f0807b46e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:55.536250 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.536188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec157b-20dc-49a1-ba71-f50cb28126b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44\" (UID: \"1dec157b-20dc-49a1-ba71-f50cb28126b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:55.536885 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.536865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b41998d-4972-420b-a6de-79f0807b46e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm\" (UID: \"6b41998d-4972-420b-a6de-79f0807b46e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:55.543021 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.542995 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44"] Sep 15 11:52:55.549822 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.549801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tt2\" (UniqueName: \"kubernetes.io/projected/6b41998d-4972-420b-a6de-79f0807b46e6-kube-api-access-t7tt2\") pod \"payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm\" (UID: \"6b41998d-4972-420b-a6de-79f0807b46e6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:55.637227 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.637161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lbf\" (UniqueName: \"kubernetes.io/projected/1dec157b-20dc-49a1-ba71-f50cb28126b0-kube-api-access-58lbf\") pod \"payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44\" (UID: \"1dec157b-20dc-49a1-ba71-f50cb28126b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:55.637386 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.637281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec157b-20dc-49a1-ba71-f50cb28126b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44\" (UID: \"1dec157b-20dc-49a1-ba71-f50cb28126b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:55.637963 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.637946 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec157b-20dc-49a1-ba71-f50cb28126b0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44\" (UID: \"1dec157b-20dc-49a1-ba71-f50cb28126b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:55.646309 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.646283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:55.676155 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.676120 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lbf\" (UniqueName: \"kubernetes.io/projected/1dec157b-20dc-49a1-ba71-f50cb28126b0-kube-api-access-58lbf\") pod \"payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44\" (UID: \"1dec157b-20dc-49a1-ba71-f50cb28126b0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:55.726635 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.726585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:55.795658 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.795625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7890-a12d54" event={"ID":"b3325f8e-21d6-4519-b994-6464410b83c3","Type":"ContainerStarted","Data":"e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3"} Sep 15 11:52:55.795968 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.795671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7890-a12d54" event={"ID":"b3325f8e-21d6-4519-b994-6464410b83c3","Type":"ContainerStarted","Data":"cee9812185ea0644b562ae1a12b773200c5f63288e6ab1808b212734fb8c7317"} Sep 15 11:52:55.797274 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.797183 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" event={"ID":"c20014d5-1a23-4a5d-b022-5a0b34192400","Type":"ContainerStarted","Data":"61b267ad8d7540593d8ee3e5e675773023c3316d101fc671c9bb548901797d07"} Sep 15 11:52:55.797377 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.797278 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7890-c59673" podUID="fa01de0b-63df-422c-b8f0-705b338bcf26" containerName="test-curl-7890-c59673" containerID="cri-o://e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30" gracePeriod=2 Sep 15 11:52:55.821478 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.821428 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7890-a12d54" podStartSLOduration=1.821410512 podStartE2EDuration="1.821410512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:55.820943014 +0000 UTC m=+863.166051621" watchObservedRunningTime="2026-09-15 11:52:55.821410512 +0000 UTC m=+863.166519115" Sep 15 11:52:55.848026 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.847956 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf"] Sep 15 11:52:55.850095 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.850080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:55.853158 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.853133 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-29d630-serving-cert\"" Sep 15 11:52:55.883420 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.883293 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf"] Sep 15 11:52:55.938765 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.938733 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm"] Sep 15 11:52:55.940192 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.940169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:55.940306 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.940287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-metrics-certs\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:55.940373 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.940330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7cd0ba-349a-48b6-9577-606aac4f8068-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:55.940373 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:55.940356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw582\" (UniqueName: \"kubernetes.io/projected/cd7cd0ba-349a-48b6-9577-606aac4f8068-kube-api-access-xw582\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:55.942153 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:55.942130 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b41998d_4972_420b_a6de_79f0807b46e6.slice/crio-f30c26fab88346b9c3a7016a7f70f870869d785df1df57d2255730db0e4bc35b WatchSource:0}: Error finding container f30c26fab88346b9c3a7016a7f70f870869d785df1df57d2255730db0e4bc35b: Status 404 returned error can't find the container with id f30c26fab88346b9c3a7016a7f70f870869d785df1df57d2255730db0e4bc35b Sep 15 11:52:56.012524 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.012495 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44"] Sep 15 11:52:56.016382 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:56.016346 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dec157b_20dc_49a1_ba71_f50cb28126b0.slice/crio-7e157b3414de60b5fd2493b6893a975b1d3f76fa7c8358a3fcc844e415b2435d WatchSource:0}: Error finding container 7e157b3414de60b5fd2493b6893a975b1d3f76fa7c8358a3fcc844e415b2435d: Status 404 returned error can't find the container with id 7e157b3414de60b5fd2493b6893a975b1d3f76fa7c8358a3fcc844e415b2435d Sep 15 11:52:56.040988 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.040959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.041126 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.041020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-metrics-certs\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.041126 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.041058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7cd0ba-349a-48b6-9577-606aac4f8068-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.041126 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.041083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw582\" (UniqueName: \"kubernetes.io/projected/cd7cd0ba-349a-48b6-9577-606aac4f8068-kube-api-access-xw582\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.041126 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:56.041109 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-serving-cert: secret "maas-api-e2e-worker-w0-29d630-serving-cert" not found Sep 15 11:52:56.041366 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:56.041173 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls podName:cd7cd0ba-349a-48b6-9577-606aac4f8068 nodeName:}" failed. No retries permitted until 2026-09-15 11:52:56.541156259 +0000 UTC m=+863.886264842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls") pod "maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" (UID: "cd7cd0ba-349a-48b6-9577-606aac4f8068") : secret "maas-api-e2e-worker-w0-29d630-serving-cert" not found Sep 15 11:52:56.041666 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.041650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7cd0ba-349a-48b6-9577-606aac4f8068-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.043731 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.043710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-metrics-certs\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.061302 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.061273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw582\" (UniqueName: \"kubernetes.io/projected/cd7cd0ba-349a-48b6-9577-606aac4f8068-kube-api-access-xw582\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.544762 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.544724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.547502 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.547474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls\") pod \"maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.644525 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.644436 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7890-a12d54"] Sep 15 11:52:56.648323 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.648249 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7890-a12d54"] Sep 15 11:52:56.764568 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.764528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:56.805336 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.805299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" event={"ID":"c20014d5-1a23-4a5d-b022-5a0b34192400","Type":"ContainerStarted","Data":"e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6"} Sep 15 11:52:56.805753 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.805400 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:52:56.807935 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.807700 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" event={"ID":"1dec157b-20dc-49a1-ba71-f50cb28126b0","Type":"ContainerStarted","Data":"1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7"} Sep 15 11:52:56.807935 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.807756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" event={"ID":"1dec157b-20dc-49a1-ba71-f50cb28126b0","Type":"ContainerStarted","Data":"7e157b3414de60b5fd2493b6893a975b1d3f76fa7c8358a3fcc844e415b2435d"} Sep 15 11:52:56.807935 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.807862 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:52:56.810965 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.810897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" event={"ID":"6b41998d-4972-420b-a6de-79f0807b46e6","Type":"ContainerStarted","Data":"2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f"} Sep 15 11:52:56.810965 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.810944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" event={"ID":"6b41998d-4972-420b-a6de-79f0807b46e6","Type":"ContainerStarted","Data":"f30c26fab88346b9c3a7016a7f70f870869d785df1df57d2255730db0e4bc35b"} Sep 15 11:52:56.811118 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.810979 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:52:56.811265 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:56.811250 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7890-a12d54" container="test-curl-7890-a12d54" Sep 15 11:52:56.891899 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.891839 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" podStartSLOduration=2.546895024 podStartE2EDuration="2.891819694s" totalImagesPullingTime="344.92467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:56.837403658 +0000 UTC m=+864.182512265" watchObservedRunningTime="2026-09-15 11:52:56.891819694 +0000 UTC m=+864.236928305" Sep 15 11:52:56.892376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.892349 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" podStartSLOduration=1.892341209 podStartE2EDuration="1.892341209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:56.890361719 +0000 UTC m=+864.235470334" watchObservedRunningTime="2026-09-15 11:52:56.892341209 +0000 UTC m=+864.237449817" Sep 15 11:52:56.907240 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.907150 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt"] Sep 15 11:52:56.907699 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.907685 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3325f8e-21d6-4519-b994-6464410b83c3" containerName="test-curl-7890-a12d54" Sep 15 11:52:56.907746 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.907703 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3325f8e-21d6-4519-b994-6464410b83c3" containerName="test-curl-7890-a12d54" Sep 15 11:52:56.907812 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.907803 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3325f8e-21d6-4519-b994-6464410b83c3" containerName="test-curl-7890-a12d54" Sep 15 11:52:56.910434 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.910415 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:56.913180 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.913153 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-29d630\"" Sep 15 11:52:56.919976 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.919954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt"] Sep 15 11:52:56.923628 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.923036 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" podStartSLOduration=1.923018059 podStartE2EDuration="1.923018059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:56.922340684 +0000 UTC m=+864.267449291" watchObservedRunningTime="2026-09-15 11:52:56.923018059 +0000 UTC m=+864.268126664" Sep 15 11:52:56.978648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.978613 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm"] Sep 15 11:52:56.983773 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:56.983726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.010276 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.010247 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-29d630-dockercfg-t68rn\"" Sep 15 11:52:57.015094 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.015069 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm"] Sep 15 11:52:57.049157 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.049126 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm\" (UID: \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.049348 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.049186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2daa27-f5c0-4458-bb7d-184d9e40a907-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt\" (UID: \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:57.049348 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.049302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk9rj\" (UniqueName: \"kubernetes.io/projected/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-kube-api-access-rk9rj\") pod \"payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm\" (UID: \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.049348 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.049340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct98f\" (UniqueName: \"kubernetes.io/projected/aa2daa27-f5c0-4458-bb7d-184d9e40a907-kube-api-access-ct98f\") pod \"payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt\" (UID: \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:57.059674 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.059631 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf"] Sep 15 11:52:57.062549 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:57.062521 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7cd0ba_349a_48b6_9577_606aac4f8068.slice/crio-f7c0cf9c3384fe5f6de60cae776d46e6cabae5dff52856c1f18d3839fa1781b1 WatchSource:0}: Error finding container f7c0cf9c3384fe5f6de60cae776d46e6cabae5dff52856c1f18d3839fa1781b1: Status 404 returned error can't find the container with id f7c0cf9c3384fe5f6de60cae776d46e6cabae5dff52856c1f18d3839fa1781b1 Sep 15 11:52:57.150242 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.150186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct98f\" (UniqueName: \"kubernetes.io/projected/aa2daa27-f5c0-4458-bb7d-184d9e40a907-kube-api-access-ct98f\") pod \"payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt\" (UID: \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:57.150430 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.150304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm\" (UID: \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.150430 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.150354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2daa27-f5c0-4458-bb7d-184d9e40a907-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt\" (UID: \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:57.150430 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.150388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk9rj\" (UniqueName: \"kubernetes.io/projected/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-kube-api-access-rk9rj\") pod \"payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm\" (UID: \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.151265 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.151239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2daa27-f5c0-4458-bb7d-184d9e40a907-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt\" (UID: \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:57.151351 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.151266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm\" (UID: \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.194419 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.194384 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct98f\" (UniqueName: \"kubernetes.io/projected/aa2daa27-f5c0-4458-bb7d-184d9e40a907-kube-api-access-ct98f\") pod \"payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt\" (UID: \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:57.194562 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.194517 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk9rj\" (UniqueName: \"kubernetes.io/projected/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-kube-api-access-rk9rj\") pod \"payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm\" (UID: \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.224224 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.224163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:57.296441 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.296173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.415054 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.415024 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9"] Sep 15 11:52:57.418897 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.418876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.433313 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.433291 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-97d02d-serving-cert\"" Sep 15 11:52:57.442722 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.441901 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9"] Sep 15 11:52:57.525804 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.525765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt"] Sep 15 11:52:57.532347 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:57.532320 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2daa27_f5c0_4458_bb7d_184d9e40a907.slice/crio-f78d3067d3ea07e5da317560ae79e80b1606454c8605462531611a991bba3a4e WatchSource:0}: Error finding container f78d3067d3ea07e5da317560ae79e80b1606454c8605462531611a991bba3a4e: Status 404 returned error can't find the container with id f78d3067d3ea07e5da317560ae79e80b1606454c8605462531611a991bba3a4e Sep 15 11:52:57.554676 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.554600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-metrics-certs\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.554676 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.554655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.554854 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.554709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7728\" (UniqueName: \"kubernetes.io/projected/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-kube-api-access-v7728\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.554854 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.554794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-maas-api-tls\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.612953 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.612369 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm"] Sep 15 11:52:57.613785 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:57.613735 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d1d26a_3430_4ee2_8135_f0fc5f4ec42e.slice/crio-f113e949ad901cf537e37ceb2a5b3997883875401346429adc3eefffcda2aa18 WatchSource:0}: Error finding container f113e949ad901cf537e37ceb2a5b3997883875401346429adc3eefffcda2aa18: Status 404 returned error can't find the container with id f113e949ad901cf537e37ceb2a5b3997883875401346429adc3eefffcda2aa18 Sep 15 11:52:57.656308 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.656267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-metrics-certs\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.656429 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.656319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.656429 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.656365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7728\" (UniqueName: \"kubernetes.io/projected/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-kube-api-access-v7728\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.656429 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.656414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-maas-api-tls\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.657457 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.657428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.659129 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.659099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-maas-api-tls\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.659248 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.659101 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-metrics-certs\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.679320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.679298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7728\" (UniqueName: \"kubernetes.io/projected/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-kube-api-access-v7728\") pod \"maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.750043 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.749947 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:57.821666 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.821429 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" event={"ID":"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e","Type":"ContainerStarted","Data":"9d89637aa585c570b0a474ba181aee11d5923fe9ca844a331e012a1639245d2c"} Sep 15 11:52:57.821666 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.821488 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" event={"ID":"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e","Type":"ContainerStarted","Data":"f113e949ad901cf537e37ceb2a5b3997883875401346429adc3eefffcda2aa18"} Sep 15 11:52:57.822189 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.822161 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:52:57.825076 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.825051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" event={"ID":"aa2daa27-f5c0-4458-bb7d-184d9e40a907","Type":"ContainerStarted","Data":"e23eeddb6cf93d291ecb415bed10fb77adfe1c4f8ca67547a06e01dfa72a623a"} Sep 15 11:52:57.825314 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.825095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" event={"ID":"aa2daa27-f5c0-4458-bb7d-184d9e40a907","Type":"ContainerStarted","Data":"f78d3067d3ea07e5da317560ae79e80b1606454c8605462531611a991bba3a4e"} Sep 15 11:52:57.825314 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.825131 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:52:57.830047 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.828140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" event={"ID":"cd7cd0ba-349a-48b6-9577-606aac4f8068","Type":"ContainerStarted","Data":"cc51485b2ac446c89572fdac8b7722f7b342d51d27abcd2292cc84b29093d9e4"} Sep 15 11:52:57.830047 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.828182 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" event={"ID":"cd7cd0ba-349a-48b6-9577-606aac4f8068","Type":"ContainerStarted","Data":"f7c0cf9c3384fe5f6de60cae776d46e6cabae5dff52856c1f18d3839fa1781b1"} Sep 15 11:52:57.830047 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.828232 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:52:57.830047 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.829066 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7890-a12d54" podUID="b3325f8e-21d6-4519-b994-6464410b83c3" containerName="test-curl-7890-a12d54" containerID="cri-o://e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3" gracePeriod=2 Sep 15 11:52:57.868411 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.868353 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" podStartSLOduration=1.868331891 podStartE2EDuration="1.868331891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:57.867361449 +0000 UTC m=+865.212470057" watchObservedRunningTime="2026-09-15 11:52:57.868331891 +0000 UTC m=+865.213440500" Sep 15 11:52:57.934533 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.930580 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" podStartSLOduration=1.9305577760000001 podStartE2EDuration="1.930557776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:57.925417076 +0000 UTC m=+865.270525686" watchObservedRunningTime="2026-09-15 11:52:57.930557776 +0000 UTC m=+865.275666384" Sep 15 11:52:57.954721 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.954668 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" podStartSLOduration=2.544118166 podStartE2EDuration="2.954643682s" totalImagesPullingTime="410.525516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:57.954064245 +0000 UTC m=+865.299172850" watchObservedRunningTime="2026-09-15 11:52:57.954643682 +0000 UTC m=+865.299752287" Sep 15 11:52:57.990442 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:57.990406 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7890-c59673" Sep 15 11:52:58.060129 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.060024 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7cwj\" (UniqueName: \"kubernetes.io/projected/fa01de0b-63df-422c-b8f0-705b338bcf26-kube-api-access-b7cwj\") pod \"fa01de0b-63df-422c-b8f0-705b338bcf26\" (UID: \"fa01de0b-63df-422c-b8f0-705b338bcf26\") " Sep 15 11:52:58.062260 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.062237 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa01de0b-63df-422c-b8f0-705b338bcf26-kube-api-access-b7cwj" (OuterVolumeSpecName: "kube-api-access-b7cwj") pod "fa01de0b-63df-422c-b8f0-705b338bcf26" (UID: "fa01de0b-63df-422c-b8f0-705b338bcf26"). InnerVolumeSpecName "kube-api-access-b7cwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:52:58.080673 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.080633 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9"] Sep 15 11:52:58.082954 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:52:58.082930 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1e4751_4c95_4e92_bb96_1934ead2ffb4.slice/crio-9349ec78833551bdf815a5ba34942ac5d39662adc35fd9a368a279a9c350bf6e WatchSource:0}: Error finding container 9349ec78833551bdf815a5ba34942ac5d39662adc35fd9a368a279a9c350bf6e: Status 404 returned error can't find the container with id 9349ec78833551bdf815a5ba34942ac5d39662adc35fd9a368a279a9c350bf6e Sep 15 11:52:58.161421 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.161384 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7cwj\" (UniqueName: \"kubernetes.io/projected/fa01de0b-63df-422c-b8f0-705b338bcf26-kube-api-access-b7cwj\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:52:58.834779 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.834678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" event={"ID":"6e1e4751-4c95-4e92-bb96-1934ead2ffb4","Type":"ContainerStarted","Data":"dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd"} Sep 15 11:52:58.834779 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.834749 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:52:58.834779 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.834772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" event={"ID":"6e1e4751-4c95-4e92-bb96-1934ead2ffb4","Type":"ContainerStarted","Data":"9349ec78833551bdf815a5ba34942ac5d39662adc35fd9a368a279a9c350bf6e"} Sep 15 11:52:58.835947 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.835921 2579 generic.go:356] "Generic (PLEG): container finished" podID="fa01de0b-63df-422c-b8f0-705b338bcf26" containerID="e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30" exitCode=137 Sep 15 11:52:58.836069 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.835958 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7890-c59673" Sep 15 11:52:58.836069 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.836015 2579 scope.go:117] "RemoveContainer" containerID="e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30" Sep 15 11:52:58.847088 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.847070 2579 scope.go:117] "RemoveContainer" containerID="e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30" Sep 15 11:52:58.847425 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:58.847405 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30\": container with ID starting with e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30 not found: ID does not exist" containerID="e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30" Sep 15 11:52:58.847501 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.847438 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30"} err="failed to get container status \"e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30\": rpc error: code = NotFound desc = could not find container \"e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30\": container with ID starting with e854e71c319979f5c8270a6243933c664b7fc8b0f3c25b83c424c445f2b64f30 not found: ID does not exist" Sep 15 11:52:58.928690 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:58.928450 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" podStartSLOduration=1.4957826 podStartE2EDuration="1.928426383s" totalImagesPullingTime="432.643783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:52:58.92704419 +0000 UTC m=+866.272152795" watchObservedRunningTime="2026-09-15 11:52:58.928426383 +0000 UTC m=+866.273534990" Sep 15 11:52:58.935243 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:52:58.935195 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7890-c59673\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="fa01de0b-63df-422c-b8f0-705b338bcf26" pod="opendatahub/test-curl-7890-c59673" Sep 15 11:52:59.189874 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:52:59.189841 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa01de0b-63df-422c-b8f0-705b338bcf26" path="/var/lib/kubelet/pods/fa01de0b-63df-422c-b8f0-705b338bcf26/volumes" Sep 15 11:53:00.025272 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.025242 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7890-a12d54" Sep 15 11:53:00.179544 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.179510 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lpmp\" (UniqueName: \"kubernetes.io/projected/b3325f8e-21d6-4519-b994-6464410b83c3-kube-api-access-2lpmp\") pod \"b3325f8e-21d6-4519-b994-6464410b83c3\" (UID: \"b3325f8e-21d6-4519-b994-6464410b83c3\") " Sep 15 11:53:00.181977 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.181950 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3325f8e-21d6-4519-b994-6464410b83c3-kube-api-access-2lpmp" (OuterVolumeSpecName: "kube-api-access-2lpmp") pod "b3325f8e-21d6-4519-b994-6464410b83c3" (UID: "b3325f8e-21d6-4519-b994-6464410b83c3"). InnerVolumeSpecName "kube-api-access-2lpmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:53:00.280519 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.280459 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lpmp\" (UniqueName: \"kubernetes.io/projected/b3325f8e-21d6-4519-b994-6464410b83c3-kube-api-access-2lpmp\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:53:00.851737 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.851696 2579 generic.go:356] "Generic (PLEG): container finished" podID="b3325f8e-21d6-4519-b994-6464410b83c3" containerID="e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3" exitCode=137 Sep 15 11:53:00.851933 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.851764 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7890-a12d54" Sep 15 11:53:00.851933 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.851798 2579 scope.go:117] "RemoveContainer" containerID="e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3" Sep 15 11:53:00.861037 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.861019 2579 scope.go:117] "RemoveContainer" containerID="e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3" Sep 15 11:53:00.861493 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:53:00.861465 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3\": container with ID starting with e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3 not found: ID does not exist" containerID="e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3" Sep 15 11:53:00.861618 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:00.861503 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3"} err="failed to get container status \"e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3\": rpc error: code = NotFound desc = could not find container \"e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3\": container with ID starting with e6537ca407e4e5134469ab55890f46f207ff62df14a6a2442f79bbca7a8c51e3 not found: ID does not exist" Sep 15 11:53:00.862374 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:53:00.862347 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7890-a12d54\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="b3325f8e-21d6-4519-b994-6464410b83c3" pod="opendatahub/test-curl-7890-a12d54" Sep 15 11:53:01.190313 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:01.190276 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3325f8e-21d6-4519-b994-6464410b83c3" path="/var/lib/kubelet/pods/b3325f8e-21d6-4519-b994-6464410b83c3/volumes" Sep 15 11:53:02.834381 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:02.834354 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 11:53:03.842950 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:03.842921 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 11:53:04.686324 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.686293 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r"] Sep 15 11:53:04.691994 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.691967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.694624 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.694604 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-b42f08-serving-cert\"" Sep 15 11:53:04.710453 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.710417 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r"] Sep 15 11:53:04.830076 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.830038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.830285 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.830108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0985f6-5034-4de8-b9fd-37d37c141bfe-openshift-service-ca\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.830285 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.830129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-metrics-certs\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.830285 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.830173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdd8k\" (UniqueName: \"kubernetes.io/projected/cc0985f6-5034-4de8-b9fd-37d37c141bfe-kube-api-access-pdd8k\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.848551 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.848523 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 11:53:04.931600 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.931567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.931779 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.931671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0985f6-5034-4de8-b9fd-37d37c141bfe-openshift-service-ca\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.931779 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.931701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-metrics-certs\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.931779 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:53:04.931730 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-serving-cert: secret "maas-api-e2e-autores-b42f08-serving-cert" not found Sep 15 11:53:04.931946 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:53:04.931814 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls podName:cc0985f6-5034-4de8-b9fd-37d37c141bfe nodeName:}" failed. No retries permitted until 2026-09-15 11:53:05.431789931 +0000 UTC m=+872.776898530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls") pod "maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" (UID: "cc0985f6-5034-4de8-b9fd-37d37c141bfe") : secret "maas-api-e2e-autores-b42f08-serving-cert" not found Sep 15 11:53:04.931946 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.931838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdd8k\" (UniqueName: \"kubernetes.io/projected/cc0985f6-5034-4de8-b9fd-37d37c141bfe-kube-api-access-pdd8k\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.932647 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.932585 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0985f6-5034-4de8-b9fd-37d37c141bfe-openshift-service-ca\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.934379 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.934357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-metrics-certs\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:04.942803 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:04.942558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdd8k\" (UniqueName: \"kubernetes.io/projected/cc0985f6-5034-4de8-b9fd-37d37c141bfe-kube-api-access-pdd8k\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:05.436450 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:05.436414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:05.436621 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:53:05.436593 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-serving-cert: secret "maas-api-e2e-autores-b42f08-serving-cert" not found Sep 15 11:53:05.436714 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:53:05.436693 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls podName:cc0985f6-5034-4de8-b9fd-37d37c141bfe nodeName:}" failed. No retries permitted until 2026-09-15 11:53:06.436669574 +0000 UTC m=+873.781778158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls") pod "maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" (UID: "cc0985f6-5034-4de8-b9fd-37d37c141bfe") : secret "maas-api-e2e-autores-b42f08-serving-cert" not found Sep 15 11:53:05.952063 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:05.952030 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk"] Sep 15 11:53:05.955965 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:05.955948 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:05.958784 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:05.958761 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b42f08-dockercfg-mhhfr\"" Sep 15 11:53:05.959119 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:05.959098 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b42f08\"" Sep 15 11:53:05.974982 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:05.974948 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk"] Sep 15 11:53:06.042530 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.042496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk\" (UID: \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\") " pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:06.042694 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.042551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rsn6\" (UniqueName: \"kubernetes.io/projected/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-kube-api-access-2rsn6\") pod \"payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk\" (UID: \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\") " pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:06.143271 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.143233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk\" (UID: \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\") " pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:06.143466 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.143310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rsn6\" (UniqueName: \"kubernetes.io/projected/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-kube-api-access-2rsn6\") pod \"payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk\" (UID: \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\") " pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:06.143918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.143899 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-plugins-config-volume\") pod \"payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk\" (UID: \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\") " pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:06.153058 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.153027 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rsn6\" (UniqueName: \"kubernetes.io/projected/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-kube-api-access-2rsn6\") pod \"payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk\" (UID: \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\") " pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:06.267242 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.267125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:06.447075 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.447039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:06.449658 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.449634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls\") pod \"maas-api-e2e-autores-b42f08-77f955fd54-kzc6r\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:06.504781 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.504743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:06.545161 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.545117 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk"] Sep 15 11:53:06.546999 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:53:06.546974 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94cbabe_95cf_44c1_bd7e_94a5c9d251c5.slice/crio-bdc5d8d9c01af6ded940588a6fe0be7531f322b8eac545c49778e2e8c298b9e3 WatchSource:0}: Error finding container bdc5d8d9c01af6ded940588a6fe0be7531f322b8eac545c49778e2e8c298b9e3: Status 404 returned error can't find the container with id bdc5d8d9c01af6ded940588a6fe0be7531f322b8eac545c49778e2e8c298b9e3 Sep 15 11:53:06.786905 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.786879 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r"] Sep 15 11:53:06.788657 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:53:06.788632 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0985f6_5034_4de8_b9fd_37d37c141bfe.slice/crio-0c31def8874999412a4773c8e9f01d08030dd082149c7b216a8ca20c83dac9e3 WatchSource:0}: Error finding container 0c31def8874999412a4773c8e9f01d08030dd082149c7b216a8ca20c83dac9e3: Status 404 returned error can't find the container with id 0c31def8874999412a4773c8e9f01d08030dd082149c7b216a8ca20c83dac9e3 Sep 15 11:53:06.889328 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.889189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" event={"ID":"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5","Type":"ContainerStarted","Data":"73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84"} Sep 15 11:53:06.889328 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.889295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" event={"ID":"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5","Type":"ContainerStarted","Data":"bdc5d8d9c01af6ded940588a6fe0be7531f322b8eac545c49778e2e8c298b9e3"} Sep 15 11:53:06.889328 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.889326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:06.890562 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.890540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" event={"ID":"cc0985f6-5034-4de8-b9fd-37d37c141bfe","Type":"ContainerStarted","Data":"0c31def8874999412a4773c8e9f01d08030dd082149c7b216a8ca20c83dac9e3"} Sep 15 11:53:06.914350 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:06.914297 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" podStartSLOduration=1.91427967 podStartE2EDuration="1.91427967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:06.912354615 +0000 UTC m=+874.257463223" watchObservedRunningTime="2026-09-15 11:53:06.91427967 +0000 UTC m=+874.259388273" Sep 15 11:53:07.830426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:07.830392 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 11:53:07.830857 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:07.830833 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 11:53:07.900026 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:07.899981 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" event={"ID":"cc0985f6-5034-4de8-b9fd-37d37c141bfe","Type":"ContainerStarted","Data":"5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2"} Sep 15 11:53:07.900255 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:07.900050 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:07.935808 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:07.935761 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" podStartSLOduration=3.5208447940000003 podStartE2EDuration="3.93574262s" totalImagesPullingTime="414.897826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:07.935477721 +0000 UTC m=+875.280586338" watchObservedRunningTime="2026-09-15 11:53:07.93574262 +0000 UTC m=+875.280851225" Sep 15 11:53:08.365171 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.365127 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt"] Sep 15 11:53:08.368832 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.368810 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.371100 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.371079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-814e2181-serving-cert\"" Sep 15 11:53:08.380130 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.380076 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt"] Sep 15 11:53:08.467589 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.467558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-metrics-certs\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.467758 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.467610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780b8343-77ec-45fb-8fed-0b7a7baaa951-openshift-service-ca\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.467758 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.467672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-maas-api-tls\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.467758 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.467745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s72dt\" (UniqueName: \"kubernetes.io/projected/780b8343-77ec-45fb-8fed-0b7a7baaa951-kube-api-access-s72dt\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.568542 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.568502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780b8343-77ec-45fb-8fed-0b7a7baaa951-openshift-service-ca\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.568713 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.568564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-maas-api-tls\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.568713 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.568645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s72dt\" (UniqueName: \"kubernetes.io/projected/780b8343-77ec-45fb-8fed-0b7a7baaa951-kube-api-access-s72dt\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.568713 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.568683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-metrics-certs\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.569365 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.569302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780b8343-77ec-45fb-8fed-0b7a7baaa951-openshift-service-ca\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.571321 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.571298 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-maas-api-tls\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.571443 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.571423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-metrics-certs\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.580798 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.580775 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s72dt\" (UniqueName: \"kubernetes.io/projected/780b8343-77ec-45fb-8fed-0b7a7baaa951-kube-api-access-s72dt\") pod \"maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.679995 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.679971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:08.838630 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.838602 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 11:53:08.961989 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:08.961961 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt"] Sep 15 11:53:08.963193 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:53:08.963169 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780b8343_77ec_45fb_8fed_0b7a7baaa951.slice/crio-9df32f3b1ce8506f6e9ad3147a21f505458c174de541c5cdedac77a069acf18d WatchSource:0}: Error finding container 9df32f3b1ce8506f6e9ad3147a21f505458c174de541c5cdedac77a069acf18d: Status 404 returned error can't find the container with id 9df32f3b1ce8506f6e9ad3147a21f505458c174de541c5cdedac77a069acf18d Sep 15 11:53:09.845527 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:09.845496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 11:53:09.915351 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:09.915318 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" event={"ID":"780b8343-77ec-45fb-8fed-0b7a7baaa951","Type":"ContainerStarted","Data":"46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741"} Sep 15 11:53:09.915516 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:09.915368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" event={"ID":"780b8343-77ec-45fb-8fed-0b7a7baaa951","Type":"ContainerStarted","Data":"9df32f3b1ce8506f6e9ad3147a21f505458c174de541c5cdedac77a069acf18d"} Sep 15 11:53:09.915516 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:09.915462 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:09.940939 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:09.940474 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" podStartSLOduration=1.53671845 podStartE2EDuration="1.940451979s" totalImagesPullingTime="403.733529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:09.939712949 +0000 UTC m=+877.284821554" watchObservedRunningTime="2026-09-15 11:53:09.940451979 +0000 UTC m=+877.285560584" Sep 15 11:53:13.913565 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:13.913536 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:53:15.927065 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:15.927037 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:17.901980 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:17.901945 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:53:21.821099 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.821055 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw"] Sep 15 11:53:21.825215 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.825173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:21.832165 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.832137 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e9bc87\"" Sep 15 11:53:21.832165 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.832137 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e9bc87-dockercfg-78xb4\"" Sep 15 11:53:21.840487 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.840460 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw"] Sep 15 11:53:21.888716 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.888685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jhnm\" (UniqueName: \"kubernetes.io/projected/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-kube-api-access-7jhnm\") pod \"payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw\" (UID: \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:21.888874 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.888731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw\" (UID: \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:21.989376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.989326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jhnm\" (UniqueName: \"kubernetes.io/projected/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-kube-api-access-7jhnm\") pod \"payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw\" (UID: \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:21.989376 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.989385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw\" (UID: \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:21.990063 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:21.990043 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw\" (UID: \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:22.003609 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:22.003575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jhnm\" (UniqueName: \"kubernetes.io/projected/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-kube-api-access-7jhnm\") pod \"payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw\" (UID: \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:22.166466 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:22.166380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:22.446884 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:22.446858 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw"] Sep 15 11:53:22.449079 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:53:22.449050 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34861cd6_e8bb_4c4c_a2bd_fa9c29472e81.slice/crio-f30601d580acb1f33dae10805d5bbf5262fa3e07193f7d2ef6e9a21a102033a0 WatchSource:0}: Error finding container f30601d580acb1f33dae10805d5bbf5262fa3e07193f7d2ef6e9a21a102033a0: Status 404 returned error can't find the container with id f30601d580acb1f33dae10805d5bbf5262fa3e07193f7d2ef6e9a21a102033a0 Sep 15 11:53:22.996904 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:22.996807 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" event={"ID":"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81","Type":"ContainerStarted","Data":"a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624"} Sep 15 11:53:22.997651 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:22.996917 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" event={"ID":"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81","Type":"ContainerStarted","Data":"f30601d580acb1f33dae10805d5bbf5262fa3e07193f7d2ef6e9a21a102033a0"} Sep 15 11:53:23.021189 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:23.021130 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" podStartSLOduration=2.021114081 podStartE2EDuration="2.021114081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:23.018051306 +0000 UTC m=+890.363159937" watchObservedRunningTime="2026-09-15 11:53:23.021114081 +0000 UTC m=+890.366222732" Sep 15 11:53:24.003445 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:24.003398 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:33.144426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:33.144395 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:53:33.145572 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:33.145549 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:53:35.011552 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.011522 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 11:53:35.446402 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.446369 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt"] Sep 15 11:53:35.446604 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.446566 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" podUID="780b8343-77ec-45fb-8fed-0b7a7baaa951" containerName="maas-api" containerID="cri-o://46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741" gracePeriod=30 Sep 15 11:53:35.703333 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.703258 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:35.828222 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.828164 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-metrics-certs\") pod \"780b8343-77ec-45fb-8fed-0b7a7baaa951\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " Sep 15 11:53:35.828397 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.828245 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s72dt\" (UniqueName: \"kubernetes.io/projected/780b8343-77ec-45fb-8fed-0b7a7baaa951-kube-api-access-s72dt\") pod \"780b8343-77ec-45fb-8fed-0b7a7baaa951\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " Sep 15 11:53:35.828397 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.828272 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780b8343-77ec-45fb-8fed-0b7a7baaa951-openshift-service-ca\") pod \"780b8343-77ec-45fb-8fed-0b7a7baaa951\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " Sep 15 11:53:35.828397 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.828289 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-maas-api-tls\") pod \"780b8343-77ec-45fb-8fed-0b7a7baaa951\" (UID: \"780b8343-77ec-45fb-8fed-0b7a7baaa951\") " Sep 15 11:53:35.828703 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.828675 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780b8343-77ec-45fb-8fed-0b7a7baaa951-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "780b8343-77ec-45fb-8fed-0b7a7baaa951" (UID: "780b8343-77ec-45fb-8fed-0b7a7baaa951"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:53:35.830766 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.830736 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "780b8343-77ec-45fb-8fed-0b7a7baaa951" (UID: "780b8343-77ec-45fb-8fed-0b7a7baaa951"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:53:35.830917 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.830822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "780b8343-77ec-45fb-8fed-0b7a7baaa951" (UID: "780b8343-77ec-45fb-8fed-0b7a7baaa951"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:53:35.830917 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.830909 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780b8343-77ec-45fb-8fed-0b7a7baaa951-kube-api-access-s72dt" (OuterVolumeSpecName: "kube-api-access-s72dt") pod "780b8343-77ec-45fb-8fed-0b7a7baaa951" (UID: "780b8343-77ec-45fb-8fed-0b7a7baaa951"). InnerVolumeSpecName "kube-api-access-s72dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:53:35.929426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.929391 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:53:35.929426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.929423 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s72dt\" (UniqueName: \"kubernetes.io/projected/780b8343-77ec-45fb-8fed-0b7a7baaa951-kube-api-access-s72dt\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:53:35.929426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.929432 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/780b8343-77ec-45fb-8fed-0b7a7baaa951-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:53:35.929644 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:35.929441 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/780b8343-77ec-45fb-8fed-0b7a7baaa951-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:53:36.075483 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.075386 2579 generic.go:356] "Generic (PLEG): container finished" podID="780b8343-77ec-45fb-8fed-0b7a7baaa951" containerID="46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741" exitCode=0 Sep 15 11:53:36.075483 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.075437 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" Sep 15 11:53:36.075483 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.075457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" event={"ID":"780b8343-77ec-45fb-8fed-0b7a7baaa951","Type":"ContainerDied","Data":"46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741"} Sep 15 11:53:36.075970 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.075509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt" event={"ID":"780b8343-77ec-45fb-8fed-0b7a7baaa951","Type":"ContainerDied","Data":"9df32f3b1ce8506f6e9ad3147a21f505458c174de541c5cdedac77a069acf18d"} Sep 15 11:53:36.075970 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.075539 2579 scope.go:117] "RemoveContainer" containerID="46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741" Sep 15 11:53:36.085353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.085335 2579 scope.go:117] "RemoveContainer" containerID="46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741" Sep 15 11:53:36.085646 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:53:36.085624 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741\": container with ID starting with 46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741 not found: ID does not exist" containerID="46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741" Sep 15 11:53:36.085708 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.085656 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741"} err="failed to get container status \"46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741\": rpc error: code = NotFound desc = could not find container \"46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741\": container with ID starting with 46b57605a8d0e5b9c3991c24b60a288601be153fcabf0d3b496681141a240741 not found: ID does not exist" Sep 15 11:53:36.099058 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.099032 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt"] Sep 15 11:53:36.110247 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:36.106598 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-814e2181-5f5868bdf5-x25bt"] Sep 15 11:53:37.189425 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:53:37.189388 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="780b8343-77ec-45fb-8fed-0b7a7baaa951" path="/var/lib/kubelet/pods/780b8343-77ec-45fb-8fed-0b7a7baaa951/volumes" Sep 15 11:55:03.603555 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.603459 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg"] Sep 15 11:55:03.604583 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.604551 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="780b8343-77ec-45fb-8fed-0b7a7baaa951" containerName="maas-api" Sep 15 11:55:03.604680 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.604614 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="780b8343-77ec-45fb-8fed-0b7a7baaa951" containerName="maas-api" Sep 15 11:55:03.604923 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.604897 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="780b8343-77ec-45fb-8fed-0b7a7baaa951" containerName="maas-api" Sep 15 11:55:03.609279 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.609251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:03.613746 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.613701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b3f1ab-4948-4b3e-8f78-912c28cda07c-plugins-config-volume\") pod \"payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg\" (UID: \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\") " pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:03.613878 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.613769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/14b3f1ab-4948-4b3e-8f78-912c28cda07c-kube-api-access-jrnnc\") pod \"payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg\" (UID: \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\") " pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:03.615557 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.615529 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b407aeb7-dockercfg-2l26d\"" Sep 15 11:55:03.616573 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.616550 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b407aeb7\"" Sep 15 11:55:03.617891 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.617867 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg"] Sep 15 11:55:03.714985 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.714949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b3f1ab-4948-4b3e-8f78-912c28cda07c-plugins-config-volume\") pod \"payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg\" (UID: \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\") " pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:03.714985 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.714986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/14b3f1ab-4948-4b3e-8f78-912c28cda07c-kube-api-access-jrnnc\") pod \"payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg\" (UID: \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\") " pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:03.715734 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.715711 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b3f1ab-4948-4b3e-8f78-912c28cda07c-plugins-config-volume\") pod \"payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg\" (UID: \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\") " pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:03.723749 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.723723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/14b3f1ab-4948-4b3e-8f78-912c28cda07c-kube-api-access-jrnnc\") pod \"payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg\" (UID: \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\") " pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:03.927782 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:03.927752 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:04.205347 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:04.205317 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg"] Sep 15 11:55:04.208055 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:55:04.208025 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b3f1ab_4948_4b3e_8f78_912c28cda07c.slice/crio-c844f0ab782338f228913458d1ca39dcfec22081d7ca5eddc2cfb10714f53069 WatchSource:0}: Error finding container c844f0ab782338f228913458d1ca39dcfec22081d7ca5eddc2cfb10714f53069: Status 404 returned error can't find the container with id c844f0ab782338f228913458d1ca39dcfec22081d7ca5eddc2cfb10714f53069 Sep 15 11:55:04.556025 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:04.555937 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" event={"ID":"14b3f1ab-4948-4b3e-8f78-912c28cda07c","Type":"ContainerStarted","Data":"dd6c660e88f3d9a6c789f2e40b13fc9ffb2e53dc8c96e50d393b5a45c6925782"} Sep 15 11:55:04.556025 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:04.555995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" event={"ID":"14b3f1ab-4948-4b3e-8f78-912c28cda07c","Type":"ContainerStarted","Data":"c844f0ab782338f228913458d1ca39dcfec22081d7ca5eddc2cfb10714f53069"} Sep 15 11:55:04.556253 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:04.556156 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:04.574183 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:04.574137 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" podStartSLOduration=1.574120935 podStartE2EDuration="1.574120935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:04.571965272 +0000 UTC m=+991.917073894" watchObservedRunningTime="2026-09-15 11:55:04.574120935 +0000 UTC m=+991.919229541" Sep 15 11:55:15.563652 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:15.563617 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:32.459618 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.459580 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg"] Sep 15 11:55:32.460088 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.459873 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" podUID="14b3f1ab-4948-4b3e-8f78-912c28cda07c" containerName="payload-processing" containerID="cri-o://dd6c660e88f3d9a6c789f2e40b13fc9ffb2e53dc8c96e50d393b5a45c6925782" gracePeriod=30 Sep 15 11:55:32.707607 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.707421 2579 generic.go:356] "Generic (PLEG): container finished" podID="14b3f1ab-4948-4b3e-8f78-912c28cda07c" containerID="dd6c660e88f3d9a6c789f2e40b13fc9ffb2e53dc8c96e50d393b5a45c6925782" exitCode=0 Sep 15 11:55:32.707607 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.707480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" event={"ID":"14b3f1ab-4948-4b3e-8f78-912c28cda07c","Type":"ContainerDied","Data":"dd6c660e88f3d9a6c789f2e40b13fc9ffb2e53dc8c96e50d393b5a45c6925782"} Sep 15 11:55:32.735126 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.735097 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:32.886937 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.886901 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b3f1ab-4948-4b3e-8f78-912c28cda07c-plugins-config-volume\") pod \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\" (UID: \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\") " Sep 15 11:55:32.887134 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.886971 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/14b3f1ab-4948-4b3e-8f78-912c28cda07c-kube-api-access-jrnnc\") pod \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\" (UID: \"14b3f1ab-4948-4b3e-8f78-912c28cda07c\") " Sep 15 11:55:32.887290 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.887266 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b3f1ab-4948-4b3e-8f78-912c28cda07c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14b3f1ab-4948-4b3e-8f78-912c28cda07c" (UID: "14b3f1ab-4948-4b3e-8f78-912c28cda07c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:55:32.889193 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.889167 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b3f1ab-4948-4b3e-8f78-912c28cda07c-kube-api-access-jrnnc" (OuterVolumeSpecName: "kube-api-access-jrnnc") pod "14b3f1ab-4948-4b3e-8f78-912c28cda07c" (UID: "14b3f1ab-4948-4b3e-8f78-912c28cda07c"). InnerVolumeSpecName "kube-api-access-jrnnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:55:32.988158 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.988073 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14b3f1ab-4948-4b3e-8f78-912c28cda07c-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:55:32.988158 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:32.988103 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnnc\" (UniqueName: \"kubernetes.io/projected/14b3f1ab-4948-4b3e-8f78-912c28cda07c-kube-api-access-jrnnc\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:55:33.715262 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:33.715236 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" Sep 15 11:55:33.715262 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:33.715237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg" event={"ID":"14b3f1ab-4948-4b3e-8f78-912c28cda07c","Type":"ContainerDied","Data":"c844f0ab782338f228913458d1ca39dcfec22081d7ca5eddc2cfb10714f53069"} Sep 15 11:55:33.715905 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:33.715307 2579 scope.go:117] "RemoveContainer" containerID="dd6c660e88f3d9a6c789f2e40b13fc9ffb2e53dc8c96e50d393b5a45c6925782" Sep 15 11:55:33.735188 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:33.735155 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg"] Sep 15 11:55:33.737325 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:33.737300 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b407aeb7-6977947c69-zgdpg"] Sep 15 11:55:35.188546 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:35.188511 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14b3f1ab-4948-4b3e-8f78-912c28cda07c" path="/var/lib/kubelet/pods/14b3f1ab-4948-4b3e-8f78-912c28cda07c/volumes" Sep 15 11:55:50.924912 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:50.924879 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r"] Sep 15 11:55:50.925540 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:50.925128 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" podUID="cc0985f6-5034-4de8-b9fd-37d37c141bfe" containerName="maas-api" containerID="cri-o://5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2" gracePeriod=30 Sep 15 11:55:50.985471 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:50.985432 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk"] Sep 15 11:55:50.985762 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:50.985703 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" podUID="e94cbabe-95cf-44c1-bd7e-94a5c9d251c5" containerName="payload-processing" containerID="cri-o://73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84" gracePeriod=30 Sep 15 11:55:51.245400 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.245370 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:55:51.251397 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.251374 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdd8k\" (UniqueName: \"kubernetes.io/projected/cc0985f6-5034-4de8-b9fd-37d37c141bfe-kube-api-access-pdd8k\") pod \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " Sep 15 11:55:51.251509 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.251412 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls\") pod \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " Sep 15 11:55:51.251509 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.251449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0985f6-5034-4de8-b9fd-37d37c141bfe-openshift-service-ca\") pod \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " Sep 15 11:55:51.251509 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.251467 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-metrics-certs\") pod \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\" (UID: \"cc0985f6-5034-4de8-b9fd-37d37c141bfe\") " Sep 15 11:55:51.251863 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.251838 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0985f6-5034-4de8-b9fd-37d37c141bfe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc0985f6-5034-4de8-b9fd-37d37c141bfe" (UID: "cc0985f6-5034-4de8-b9fd-37d37c141bfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:55:51.254347 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.254320 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0985f6-5034-4de8-b9fd-37d37c141bfe-kube-api-access-pdd8k" (OuterVolumeSpecName: "kube-api-access-pdd8k") pod "cc0985f6-5034-4de8-b9fd-37d37c141bfe" (UID: "cc0985f6-5034-4de8-b9fd-37d37c141bfe"). InnerVolumeSpecName "kube-api-access-pdd8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:55:51.254443 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.254338 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc0985f6-5034-4de8-b9fd-37d37c141bfe" (UID: "cc0985f6-5034-4de8-b9fd-37d37c141bfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:55:51.254443 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.254392 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc0985f6-5034-4de8-b9fd-37d37c141bfe" (UID: "cc0985f6-5034-4de8-b9fd-37d37c141bfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:55:51.283502 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.283468 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:55:51.352937 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.352908 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rsn6\" (UniqueName: \"kubernetes.io/projected/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-kube-api-access-2rsn6\") pod \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\" (UID: \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\") " Sep 15 11:55:51.353100 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.352953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-plugins-config-volume\") pod \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\" (UID: \"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5\") " Sep 15 11:55:51.353142 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.353112 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdd8k\" (UniqueName: \"kubernetes.io/projected/cc0985f6-5034-4de8-b9fd-37d37c141bfe-kube-api-access-pdd8k\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:55:51.353142 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.353122 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:55:51.353142 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.353130 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0985f6-5034-4de8-b9fd-37d37c141bfe-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:55:51.353142 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.353139 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc0985f6-5034-4de8-b9fd-37d37c141bfe-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:55:51.353413 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.353387 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e94cbabe-95cf-44c1-bd7e-94a5c9d251c5" (UID: "e94cbabe-95cf-44c1-bd7e-94a5c9d251c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:55:51.355280 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.355254 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-kube-api-access-2rsn6" (OuterVolumeSpecName: "kube-api-access-2rsn6") pod "e94cbabe-95cf-44c1-bd7e-94a5c9d251c5" (UID: "e94cbabe-95cf-44c1-bd7e-94a5c9d251c5"). InnerVolumeSpecName "kube-api-access-2rsn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:55:51.454580 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.454547 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rsn6\" (UniqueName: \"kubernetes.io/projected/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-kube-api-access-2rsn6\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:55:51.454580 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.454575 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:55:51.811729 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.811699 2579 generic.go:356] "Generic (PLEG): container finished" podID="e94cbabe-95cf-44c1-bd7e-94a5c9d251c5" containerID="73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84" exitCode=0 Sep 15 11:55:51.811890 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.811742 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" Sep 15 11:55:51.811890 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.811781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" event={"ID":"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5","Type":"ContainerDied","Data":"73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84"} Sep 15 11:55:51.811890 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.811831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk" event={"ID":"e94cbabe-95cf-44c1-bd7e-94a5c9d251c5","Type":"ContainerDied","Data":"bdc5d8d9c01af6ded940588a6fe0be7531f322b8eac545c49778e2e8c298b9e3"} Sep 15 11:55:51.811890 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.811853 2579 scope.go:117] "RemoveContainer" containerID="73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84" Sep 15 11:55:51.813290 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.813267 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc0985f6-5034-4de8-b9fd-37d37c141bfe" containerID="5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2" exitCode=0 Sep 15 11:55:51.813421 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.813302 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" Sep 15 11:55:51.813421 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.813322 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" event={"ID":"cc0985f6-5034-4de8-b9fd-37d37c141bfe","Type":"ContainerDied","Data":"5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2"} Sep 15 11:55:51.813421 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.813361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r" event={"ID":"cc0985f6-5034-4de8-b9fd-37d37c141bfe","Type":"ContainerDied","Data":"0c31def8874999412a4773c8e9f01d08030dd082149c7b216a8ca20c83dac9e3"} Sep 15 11:55:51.823830 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.823754 2579 scope.go:117] "RemoveContainer" containerID="73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84" Sep 15 11:55:51.825112 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:55:51.825071 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84\": container with ID starting with 73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84 not found: ID does not exist" containerID="73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84" Sep 15 11:55:51.825320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.825108 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84"} err="failed to get container status \"73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84\": rpc error: code = NotFound desc = could not find container \"73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84\": container with ID starting with 73ed72cd48e64b2c62407fb0eeab6aadbd4d43d308864b6956cc25b3df22aa84 not found: ID does not exist" Sep 15 11:55:51.825320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.825128 2579 scope.go:117] "RemoveContainer" containerID="5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2" Sep 15 11:55:51.833910 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.833888 2579 scope.go:117] "RemoveContainer" containerID="5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2" Sep 15 11:55:51.834241 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:55:51.834217 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2\": container with ID starting with 5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2 not found: ID does not exist" containerID="5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2" Sep 15 11:55:51.834320 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.834249 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2"} err="failed to get container status \"5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2\": rpc error: code = NotFound desc = could not find container \"5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2\": container with ID starting with 5eab6042d0feeaa6298e8e7a28735cb364aeeefb57080e7a891473e6fecc1be2 not found: ID does not exist" Sep 15 11:55:51.835808 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.835789 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk"] Sep 15 11:55:51.839946 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.839923 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b42f08-68dccddc7c-4rhwk"] Sep 15 11:55:51.851229 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.851185 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r"] Sep 15 11:55:51.855546 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:51.855525 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-b42f08-77f955fd54-kzc6r"] Sep 15 11:55:53.188176 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:53.188148 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0985f6-5034-4de8-b9fd-37d37c141bfe" path="/var/lib/kubelet/pods/cc0985f6-5034-4de8-b9fd-37d37c141bfe/volumes" Sep 15 11:55:53.188564 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:55:53.188497 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e94cbabe-95cf-44c1-bd7e-94a5c9d251c5" path="/var/lib/kubelet/pods/e94cbabe-95cf-44c1-bd7e-94a5c9d251c5/volumes" Sep 15 11:56:05.178607 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.178576 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr"] Sep 15 11:56:05.179151 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179125 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0985f6-5034-4de8-b9fd-37d37c141bfe" containerName="maas-api" Sep 15 11:56:05.179151 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179147 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0985f6-5034-4de8-b9fd-37d37c141bfe" containerName="maas-api" Sep 15 11:56:05.179340 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179171 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14b3f1ab-4948-4b3e-8f78-912c28cda07c" containerName="payload-processing" Sep 15 11:56:05.179340 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179180 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="14b3f1ab-4948-4b3e-8f78-912c28cda07c" containerName="payload-processing" Sep 15 11:56:05.179340 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179247 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94cbabe-95cf-44c1-bd7e-94a5c9d251c5" containerName="payload-processing" Sep 15 11:56:05.179340 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179257 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94cbabe-95cf-44c1-bd7e-94a5c9d251c5" containerName="payload-processing" Sep 15 11:56:05.179550 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179361 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94cbabe-95cf-44c1-bd7e-94a5c9d251c5" containerName="payload-processing" Sep 15 11:56:05.179550 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179375 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="14b3f1ab-4948-4b3e-8f78-912c28cda07c" containerName="payload-processing" Sep 15 11:56:05.179550 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.179385 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0985f6-5034-4de8-b9fd-37d37c141bfe" containerName="maas-api" Sep 15 11:56:05.182157 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.182132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.184288 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.184266 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-2ff8fe-serving-cert\"" Sep 15 11:56:05.192646 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.192220 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr"] Sep 15 11:56:05.280389 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.280356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.280389 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.280392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-metrics-certs\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.280634 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.280437 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df23bdf4-9329-4cae-a665-e81bdf2b38be-openshift-service-ca\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.280634 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.280590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bf7p\" (UniqueName: \"kubernetes.io/projected/df23bdf4-9329-4cae-a665-e81bdf2b38be-kube-api-access-4bf7p\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.381417 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.381378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bf7p\" (UniqueName: \"kubernetes.io/projected/df23bdf4-9329-4cae-a665-e81bdf2b38be-kube-api-access-4bf7p\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.381626 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.381444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.381626 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.381461 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-metrics-certs\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.381626 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.381489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df23bdf4-9329-4cae-a665-e81bdf2b38be-openshift-service-ca\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.381626 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:56:05.381590 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-serving-cert: secret "maas-api-e2e-explicit-2ff8fe-serving-cert" not found Sep 15 11:56:05.381846 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:56:05.381649 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls podName:df23bdf4-9329-4cae-a665-e81bdf2b38be nodeName:}" failed. No retries permitted until 2026-09-15 11:56:05.881633941 +0000 UTC m=+1053.226742524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls") pod "maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" (UID: "df23bdf4-9329-4cae-a665-e81bdf2b38be") : secret "maas-api-e2e-explicit-2ff8fe-serving-cert" not found Sep 15 11:56:05.382193 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.382175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df23bdf4-9329-4cae-a665-e81bdf2b38be-openshift-service-ca\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.384256 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.384237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-metrics-certs\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.389924 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.389904 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bf7p\" (UniqueName: \"kubernetes.io/projected/df23bdf4-9329-4cae-a665-e81bdf2b38be-kube-api-access-4bf7p\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.842095 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.842061 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj"] Sep 15 11:56:05.845922 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.845899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:05.848306 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.848280 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2ff8fe-dockercfg-9f6sh\"" Sep 15 11:56:05.848416 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.848295 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2ff8fe\"" Sep 15 11:56:05.854288 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.854270 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj"] Sep 15 11:56:05.885765 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.885734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.885948 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.885871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc676\" (UniqueName: \"kubernetes.io/projected/4f6aeb71-7554-4b1f-b513-94481d491812-kube-api-access-tc676\") pod \"payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj\" (UID: \"4f6aeb71-7554-4b1f-b513-94481d491812\") " pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:05.885948 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.885922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6aeb71-7554-4b1f-b513-94481d491812-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj\" (UID: \"4f6aeb71-7554-4b1f-b513-94481d491812\") " pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:05.888439 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.888407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls\") pod \"maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:05.986772 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.986732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6aeb71-7554-4b1f-b513-94481d491812-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj\" (UID: \"4f6aeb71-7554-4b1f-b513-94481d491812\") " pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:05.986937 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.986868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc676\" (UniqueName: \"kubernetes.io/projected/4f6aeb71-7554-4b1f-b513-94481d491812-kube-api-access-tc676\") pod \"payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj\" (UID: \"4f6aeb71-7554-4b1f-b513-94481d491812\") " pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:05.987588 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.987569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6aeb71-7554-4b1f-b513-94481d491812-plugins-config-volume\") pod \"payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj\" (UID: \"4f6aeb71-7554-4b1f-b513-94481d491812\") " pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:05.995154 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:05.995126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc676\" (UniqueName: \"kubernetes.io/projected/4f6aeb71-7554-4b1f-b513-94481d491812-kube-api-access-tc676\") pod \"payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj\" (UID: \"4f6aeb71-7554-4b1f-b513-94481d491812\") " pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:06.095978 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.095889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:06.158483 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.158382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:06.371007 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.370979 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr"] Sep 15 11:56:06.373245 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:56:06.373212 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf23bdf4_9329_4cae_a665_e81bdf2b38be.slice/crio-0ba9b8e6c793c9d91cb91acb2940ddcd74e84b734126005554207d9140e74e6c WatchSource:0}: Error finding container 0ba9b8e6c793c9d91cb91acb2940ddcd74e84b734126005554207d9140e74e6c: Status 404 returned error can't find the container with id 0ba9b8e6c793c9d91cb91acb2940ddcd74e84b734126005554207d9140e74e6c Sep 15 11:56:06.374682 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.374666 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 11:56:06.440017 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.439993 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj"] Sep 15 11:56:06.441825 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:56:06.441798 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6aeb71_7554_4b1f_b513_94481d491812.slice/crio-2efba18ca6e6c3d1c5a02de199ec73a3185ff8a89bc47751b243b88d2e147d7e WatchSource:0}: Error finding container 2efba18ca6e6c3d1c5a02de199ec73a3185ff8a89bc47751b243b88d2e147d7e: Status 404 returned error can't find the container with id 2efba18ca6e6c3d1c5a02de199ec73a3185ff8a89bc47751b243b88d2e147d7e Sep 15 11:56:06.896867 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.896832 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" event={"ID":"4f6aeb71-7554-4b1f-b513-94481d491812","Type":"ContainerStarted","Data":"9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d"} Sep 15 11:56:06.896994 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.896902 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" event={"ID":"4f6aeb71-7554-4b1f-b513-94481d491812","Type":"ContainerStarted","Data":"2efba18ca6e6c3d1c5a02de199ec73a3185ff8a89bc47751b243b88d2e147d7e"} Sep 15 11:56:06.896994 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.896932 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:56:06.899014 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.898986 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" event={"ID":"df23bdf4-9329-4cae-a665-e81bdf2b38be","Type":"ContainerStarted","Data":"0ba9b8e6c793c9d91cb91acb2940ddcd74e84b734126005554207d9140e74e6c"} Sep 15 11:56:06.914473 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:06.914419 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" podStartSLOduration=1.914399504 podStartE2EDuration="1.914399504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:06.91364189 +0000 UTC m=+1054.258750497" watchObservedRunningTime="2026-09-15 11:56:06.914399504 +0000 UTC m=+1054.259508111" Sep 15 11:56:07.905837 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:07.905794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" event={"ID":"df23bdf4-9329-4cae-a665-e81bdf2b38be","Type":"ContainerStarted","Data":"081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe"} Sep 15 11:56:07.906253 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:07.905871 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:07.925016 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:07.924970 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" podStartSLOduration=2.460786005 podStartE2EDuration="2.924955616s" totalImagesPullingTime="464.169611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:07.923639495 +0000 UTC m=+1055.268748103" watchObservedRunningTime="2026-09-15 11:56:07.924955616 +0000 UTC m=+1055.270064222" Sep 15 11:56:13.918569 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:13.918542 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:56:17.908346 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:56:17.908316 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:57:25.003293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.003248 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj"] Sep 15 11:57:25.007193 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.007171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:25.009601 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.009582 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cdf667d8\"" Sep 15 11:57:25.009773 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.009664 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cdf667d8-dockercfg-wtndr\"" Sep 15 11:57:25.015669 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.015645 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj"] Sep 15 11:57:25.035963 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.035933 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp"] Sep 15 11:57:25.039648 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.039627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:25.050564 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.050540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp"] Sep 15 11:57:25.109555 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.109517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a93599-a966-4dbe-b70f-1bf32b8a33e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj\" (UID: \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:25.109732 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.109567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjv6\" (UniqueName: \"kubernetes.io/projected/03a93599-a966-4dbe-b70f-1bf32b8a33e9-kube-api-access-lqjv6\") pod \"payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj\" (UID: \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:25.210820 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.210787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5s2x\" (UniqueName: \"kubernetes.io/projected/30a5fcc9-b549-45c2-9fe6-820968b8b614-kube-api-access-w5s2x\") pod \"payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp\" (UID: \"30a5fcc9-b549-45c2-9fe6-820968b8b614\") " pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:25.211014 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.210831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a93599-a966-4dbe-b70f-1bf32b8a33e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj\" (UID: \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:25.211014 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.210863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjv6\" (UniqueName: \"kubernetes.io/projected/03a93599-a966-4dbe-b70f-1bf32b8a33e9-kube-api-access-lqjv6\") pod \"payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj\" (UID: \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:25.211014 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.210906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5fcc9-b549-45c2-9fe6-820968b8b614-plugins-config-volume\") pod \"payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp\" (UID: \"30a5fcc9-b549-45c2-9fe6-820968b8b614\") " pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:25.211576 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.211559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a93599-a966-4dbe-b70f-1bf32b8a33e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj\" (UID: \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:25.219930 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.219906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjv6\" (UniqueName: \"kubernetes.io/projected/03a93599-a966-4dbe-b70f-1bf32b8a33e9-kube-api-access-lqjv6\") pod \"payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj\" (UID: \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:25.311523 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.311431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5s2x\" (UniqueName: \"kubernetes.io/projected/30a5fcc9-b549-45c2-9fe6-820968b8b614-kube-api-access-w5s2x\") pod \"payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp\" (UID: \"30a5fcc9-b549-45c2-9fe6-820968b8b614\") " pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:25.311523 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.311486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5fcc9-b549-45c2-9fe6-820968b8b614-plugins-config-volume\") pod \"payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp\" (UID: \"30a5fcc9-b549-45c2-9fe6-820968b8b614\") " pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:25.312299 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.312279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5fcc9-b549-45c2-9fe6-820968b8b614-plugins-config-volume\") pod \"payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp\" (UID: \"30a5fcc9-b549-45c2-9fe6-820968b8b614\") " pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:25.319024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.318999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:25.320304 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.320286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5s2x\" (UniqueName: \"kubernetes.io/projected/30a5fcc9-b549-45c2-9fe6-820968b8b614-kube-api-access-w5s2x\") pod \"payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp\" (UID: \"30a5fcc9-b549-45c2-9fe6-820968b8b614\") " pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:25.350280 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.350250 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:25.596983 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.596929 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj"] Sep 15 11:57:25.635857 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:25.635837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp"] Sep 15 11:57:25.638109 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:57:25.638081 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a5fcc9_b549_45c2_9fe6_820968b8b614.slice/crio-86531a18493bcddbed608cfe641db95f4166f41a2578dafe76e3f8150b334d05 WatchSource:0}: Error finding container 86531a18493bcddbed608cfe641db95f4166f41a2578dafe76e3f8150b334d05: Status 404 returned error can't find the container with id 86531a18493bcddbed608cfe641db95f4166f41a2578dafe76e3f8150b334d05 Sep 15 11:57:26.322477 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:26.322439 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" event={"ID":"30a5fcc9-b549-45c2-9fe6-820968b8b614","Type":"ContainerStarted","Data":"9c696dd95b60dd8fdb0af4a8e98420313e9866d54ef46e3e4ef77867835a6b5e"} Sep 15 11:57:26.322952 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:26.322503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" event={"ID":"30a5fcc9-b549-45c2-9fe6-820968b8b614","Type":"ContainerStarted","Data":"86531a18493bcddbed608cfe641db95f4166f41a2578dafe76e3f8150b334d05"} Sep 15 11:57:26.322952 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:26.322553 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:26.324110 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:26.324088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" event={"ID":"03a93599-a966-4dbe-b70f-1bf32b8a33e9","Type":"ContainerStarted","Data":"2e30c7d925516dbef1dc72ab4f06f73a4f71c1e18220f97c050fcd531566915d"} Sep 15 11:57:26.324236 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:26.324136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" event={"ID":"03a93599-a966-4dbe-b70f-1bf32b8a33e9","Type":"ContainerStarted","Data":"de7465a2cda1c3e94f0058feab7d3a1742c29f6dc77dbc055b976b4b1d1a974e"} Sep 15 11:57:26.324236 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:26.324170 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:26.340790 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:26.340740 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" podStartSLOduration=1.340722012 podStartE2EDuration="1.340722012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:26.339595639 +0000 UTC m=+1133.684704245" watchObservedRunningTime="2026-09-15 11:57:26.340722012 +0000 UTC m=+1133.685830621" Sep 15 11:57:26.356343 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:26.356304 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" podStartSLOduration=2.356289424 podStartE2EDuration="2.356289424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:26.355368491 +0000 UTC m=+1133.700477098" watchObservedRunningTime="2026-09-15 11:57:26.356289424 +0000 UTC m=+1133.701398067" Sep 15 11:57:37.331924 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:37.331889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:57:37.332467 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:37.331971 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:57:50.064818 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.064784 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr"] Sep 15 11:57:50.065333 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.065065 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" podUID="df23bdf4-9329-4cae-a665-e81bdf2b38be" containerName="maas-api" containerID="cri-o://081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe" gracePeriod=30 Sep 15 11:57:50.147768 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.147735 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj"] Sep 15 11:57:50.147967 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.147944 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" podUID="4f6aeb71-7554-4b1f-b513-94481d491812" containerName="payload-processing" containerID="cri-o://9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d" gracePeriod=30 Sep 15 11:57:50.408256 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.406480 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:57:50.443631 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.443600 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:57:50.454907 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.454882 2579 generic.go:356] "Generic (PLEG): container finished" podID="df23bdf4-9329-4cae-a665-e81bdf2b38be" containerID="081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe" exitCode=0 Sep 15 11:57:50.455046 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.454927 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" Sep 15 11:57:50.455046 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.454977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" event={"ID":"df23bdf4-9329-4cae-a665-e81bdf2b38be","Type":"ContainerDied","Data":"081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe"} Sep 15 11:57:50.455138 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.455047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr" event={"ID":"df23bdf4-9329-4cae-a665-e81bdf2b38be","Type":"ContainerDied","Data":"0ba9b8e6c793c9d91cb91acb2940ddcd74e84b734126005554207d9140e74e6c"} Sep 15 11:57:50.455138 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.455081 2579 scope.go:117] "RemoveContainer" containerID="081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe" Sep 15 11:57:50.456736 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.456710 2579 generic.go:356] "Generic (PLEG): container finished" podID="4f6aeb71-7554-4b1f-b513-94481d491812" containerID="9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d" exitCode=0 Sep 15 11:57:50.456865 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.456763 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" Sep 15 11:57:50.456865 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.456778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" event={"ID":"4f6aeb71-7554-4b1f-b513-94481d491812","Type":"ContainerDied","Data":"9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d"} Sep 15 11:57:50.456865 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.456830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj" event={"ID":"4f6aeb71-7554-4b1f-b513-94481d491812","Type":"ContainerDied","Data":"2efba18ca6e6c3d1c5a02de199ec73a3185ff8a89bc47751b243b88d2e147d7e"} Sep 15 11:57:50.471244 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.471191 2579 scope.go:117] "RemoveContainer" containerID="081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe" Sep 15 11:57:50.471680 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:57:50.471656 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe\": container with ID starting with 081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe not found: ID does not exist" containerID="081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe" Sep 15 11:57:50.471764 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.471692 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe"} err="failed to get container status \"081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe\": rpc error: code = NotFound desc = could not find container \"081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe\": container with ID starting with 081a37a113c74877a70e0ffe778dc6ac76b3c5097e590d68519425739d569dbe not found: ID does not exist" Sep 15 11:57:50.471764 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.471714 2579 scope.go:117] "RemoveContainer" containerID="9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d" Sep 15 11:57:50.483049 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.483023 2579 scope.go:117] "RemoveContainer" containerID="9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d" Sep 15 11:57:50.483368 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:57:50.483350 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d\": container with ID starting with 9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d not found: ID does not exist" containerID="9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d" Sep 15 11:57:50.483435 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.483376 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d"} err="failed to get container status \"9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d\": rpc error: code = NotFound desc = could not find container \"9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d\": container with ID starting with 9fa54f23f6c684aa5284d462ea9b697392f84937a9146aaf405d7511fcfbe30d not found: ID does not exist" Sep 15 11:57:50.534876 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.534846 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc676\" (UniqueName: \"kubernetes.io/projected/4f6aeb71-7554-4b1f-b513-94481d491812-kube-api-access-tc676\") pod \"4f6aeb71-7554-4b1f-b513-94481d491812\" (UID: \"4f6aeb71-7554-4b1f-b513-94481d491812\") " Sep 15 11:57:50.534876 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.534878 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-metrics-certs\") pod \"df23bdf4-9329-4cae-a665-e81bdf2b38be\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " Sep 15 11:57:50.535091 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.534897 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bf7p\" (UniqueName: \"kubernetes.io/projected/df23bdf4-9329-4cae-a665-e81bdf2b38be-kube-api-access-4bf7p\") pod \"df23bdf4-9329-4cae-a665-e81bdf2b38be\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " Sep 15 11:57:50.535091 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.534932 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df23bdf4-9329-4cae-a665-e81bdf2b38be-openshift-service-ca\") pod \"df23bdf4-9329-4cae-a665-e81bdf2b38be\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " Sep 15 11:57:50.535091 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.535000 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls\") pod \"df23bdf4-9329-4cae-a665-e81bdf2b38be\" (UID: \"df23bdf4-9329-4cae-a665-e81bdf2b38be\") " Sep 15 11:57:50.535091 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.535041 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6aeb71-7554-4b1f-b513-94481d491812-plugins-config-volume\") pod \"4f6aeb71-7554-4b1f-b513-94481d491812\" (UID: \"4f6aeb71-7554-4b1f-b513-94481d491812\") " Sep 15 11:57:50.535379 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.535351 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df23bdf4-9329-4cae-a665-e81bdf2b38be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df23bdf4-9329-4cae-a665-e81bdf2b38be" (UID: "df23bdf4-9329-4cae-a665-e81bdf2b38be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:57:50.535469 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.535448 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df23bdf4-9329-4cae-a665-e81bdf2b38be-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:57:50.535761 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.535483 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6aeb71-7554-4b1f-b513-94481d491812-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f6aeb71-7554-4b1f-b513-94481d491812" (UID: "4f6aeb71-7554-4b1f-b513-94481d491812"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:57:50.537270 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.537226 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6aeb71-7554-4b1f-b513-94481d491812-kube-api-access-tc676" (OuterVolumeSpecName: "kube-api-access-tc676") pod "4f6aeb71-7554-4b1f-b513-94481d491812" (UID: "4f6aeb71-7554-4b1f-b513-94481d491812"). InnerVolumeSpecName "kube-api-access-tc676". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:57:50.537270 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.537229 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df23bdf4-9329-4cae-a665-e81bdf2b38be" (UID: "df23bdf4-9329-4cae-a665-e81bdf2b38be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:57:50.537386 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.537283 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df23bdf4-9329-4cae-a665-e81bdf2b38be" (UID: "df23bdf4-9329-4cae-a665-e81bdf2b38be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 11:57:50.537386 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.537305 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df23bdf4-9329-4cae-a665-e81bdf2b38be-kube-api-access-4bf7p" (OuterVolumeSpecName: "kube-api-access-4bf7p") pod "df23bdf4-9329-4cae-a665-e81bdf2b38be" (UID: "df23bdf4-9329-4cae-a665-e81bdf2b38be"). InnerVolumeSpecName "kube-api-access-4bf7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:57:50.636144 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.636056 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc676\" (UniqueName: \"kubernetes.io/projected/4f6aeb71-7554-4b1f-b513-94481d491812-kube-api-access-tc676\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:57:50.636144 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.636088 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:57:50.636144 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.636106 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bf7p\" (UniqueName: \"kubernetes.io/projected/df23bdf4-9329-4cae-a665-e81bdf2b38be-kube-api-access-4bf7p\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:57:50.636144 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.636115 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df23bdf4-9329-4cae-a665-e81bdf2b38be-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:57:50.636144 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.636124 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f6aeb71-7554-4b1f-b513-94481d491812-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:57:50.782509 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.782483 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr"] Sep 15 11:57:50.787254 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.787172 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2ff8fe-778d58fbcd-p97fr"] Sep 15 11:57:50.797285 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.797260 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj"] Sep 15 11:57:50.804366 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:50.804346 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-2ff8fe-6c5fbc96d5-gw9bj"] Sep 15 11:57:51.188415 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:51.188388 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6aeb71-7554-4b1f-b513-94481d491812" path="/var/lib/kubelet/pods/4f6aeb71-7554-4b1f-b513-94481d491812/volumes" Sep 15 11:57:51.188777 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:57:51.188743 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df23bdf4-9329-4cae-a665-e81bdf2b38be" path="/var/lib/kubelet/pods/df23bdf4-9329-4cae-a665-e81bdf2b38be/volumes" Sep 15 11:58:04.249278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.245085 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp"] Sep 15 11:58:04.249278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.245411 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" podUID="30a5fcc9-b549-45c2-9fe6-820968b8b614" containerName="payload-processing" containerID="cri-o://9c696dd95b60dd8fdb0af4a8e98420313e9866d54ef46e3e4ef77867835a6b5e" gracePeriod=30 Sep 15 11:58:04.249278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.247653 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj"] Sep 15 11:58:04.249278 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.247848 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" podUID="03a93599-a966-4dbe-b70f-1bf32b8a33e9" containerName="payload-pre-processing" containerID="cri-o://2e30c7d925516dbef1dc72ab4f06f73a4f71c1e18220f97c050fcd531566915d" gracePeriod=30 Sep 15 11:58:04.535325 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.535296 2579 generic.go:356] "Generic (PLEG): container finished" podID="30a5fcc9-b549-45c2-9fe6-820968b8b614" containerID="9c696dd95b60dd8fdb0af4a8e98420313e9866d54ef46e3e4ef77867835a6b5e" exitCode=0 Sep 15 11:58:04.535441 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.535377 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" event={"ID":"30a5fcc9-b549-45c2-9fe6-820968b8b614","Type":"ContainerDied","Data":"9c696dd95b60dd8fdb0af4a8e98420313e9866d54ef46e3e4ef77867835a6b5e"} Sep 15 11:58:04.535507 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.535446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" event={"ID":"30a5fcc9-b549-45c2-9fe6-820968b8b614","Type":"ContainerDied","Data":"86531a18493bcddbed608cfe641db95f4166f41a2578dafe76e3f8150b334d05"} Sep 15 11:58:04.535507 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.535475 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86531a18493bcddbed608cfe641db95f4166f41a2578dafe76e3f8150b334d05" Sep 15 11:58:04.536918 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.536892 2579 generic.go:356] "Generic (PLEG): container finished" podID="03a93599-a966-4dbe-b70f-1bf32b8a33e9" containerID="2e30c7d925516dbef1dc72ab4f06f73a4f71c1e18220f97c050fcd531566915d" exitCode=0 Sep 15 11:58:04.537019 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.536931 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" event={"ID":"03a93599-a966-4dbe-b70f-1bf32b8a33e9","Type":"ContainerDied","Data":"2e30c7d925516dbef1dc72ab4f06f73a4f71c1e18220f97c050fcd531566915d"} Sep 15 11:58:04.549496 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.549474 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:58:04.553629 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.553607 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:58:04.662871 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.662838 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqjv6\" (UniqueName: \"kubernetes.io/projected/03a93599-a966-4dbe-b70f-1bf32b8a33e9-kube-api-access-lqjv6\") pod \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\" (UID: \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\") " Sep 15 11:58:04.663056 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.662886 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5fcc9-b549-45c2-9fe6-820968b8b614-plugins-config-volume\") pod \"30a5fcc9-b549-45c2-9fe6-820968b8b614\" (UID: \"30a5fcc9-b549-45c2-9fe6-820968b8b614\") " Sep 15 11:58:04.663056 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.662960 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5s2x\" (UniqueName: \"kubernetes.io/projected/30a5fcc9-b549-45c2-9fe6-820968b8b614-kube-api-access-w5s2x\") pod \"30a5fcc9-b549-45c2-9fe6-820968b8b614\" (UID: \"30a5fcc9-b549-45c2-9fe6-820968b8b614\") " Sep 15 11:58:04.663056 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.663026 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a93599-a966-4dbe-b70f-1bf32b8a33e9-plugins-config-volume\") pod \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\" (UID: \"03a93599-a966-4dbe-b70f-1bf32b8a33e9\") " Sep 15 11:58:04.663353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.663325 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a5fcc9-b549-45c2-9fe6-820968b8b614-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30a5fcc9-b549-45c2-9fe6-820968b8b614" (UID: "30a5fcc9-b549-45c2-9fe6-820968b8b614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:58:04.663483 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.663427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a93599-a966-4dbe-b70f-1bf32b8a33e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03a93599-a966-4dbe-b70f-1bf32b8a33e9" (UID: "03a93599-a966-4dbe-b70f-1bf32b8a33e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 11:58:04.665226 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.665187 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a93599-a966-4dbe-b70f-1bf32b8a33e9-kube-api-access-lqjv6" (OuterVolumeSpecName: "kube-api-access-lqjv6") pod "03a93599-a966-4dbe-b70f-1bf32b8a33e9" (UID: "03a93599-a966-4dbe-b70f-1bf32b8a33e9"). InnerVolumeSpecName "kube-api-access-lqjv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:58:04.665290 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.665237 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a5fcc9-b549-45c2-9fe6-820968b8b614-kube-api-access-w5s2x" (OuterVolumeSpecName: "kube-api-access-w5s2x") pod "30a5fcc9-b549-45c2-9fe6-820968b8b614" (UID: "30a5fcc9-b549-45c2-9fe6-820968b8b614"). InnerVolumeSpecName "kube-api-access-w5s2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:58:04.764471 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.764425 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5fcc9-b549-45c2-9fe6-820968b8b614-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:58:04.764471 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.764464 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5s2x\" (UniqueName: \"kubernetes.io/projected/30a5fcc9-b549-45c2-9fe6-820968b8b614-kube-api-access-w5s2x\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:58:04.764471 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.764475 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a93599-a966-4dbe-b70f-1bf32b8a33e9-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:58:04.764471 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:04.764484 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjv6\" (UniqueName: \"kubernetes.io/projected/03a93599-a966-4dbe-b70f-1bf32b8a33e9-kube-api-access-lqjv6\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:58:05.551510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:05.551428 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" Sep 15 11:58:05.551510 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:05.551454 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj" event={"ID":"03a93599-a966-4dbe-b70f-1bf32b8a33e9","Type":"ContainerDied","Data":"de7465a2cda1c3e94f0058feab7d3a1742c29f6dc77dbc055b976b4b1d1a974e"} Sep 15 11:58:05.552032 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:05.551538 2579 scope.go:117] "RemoveContainer" containerID="2e30c7d925516dbef1dc72ab4f06f73a4f71c1e18220f97c050fcd531566915d" Sep 15 11:58:05.552032 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:05.551667 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp" Sep 15 11:58:05.568807 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:05.568785 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp"] Sep 15 11:58:05.575707 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:05.575683 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cdf667d8-76cf8f97b5-qf7wp"] Sep 15 11:58:05.586951 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:05.586928 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj"] Sep 15 11:58:05.589009 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:05.588991 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cdf667d8-6bbf4c56c8-qm4zj"] Sep 15 11:58:07.189241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:07.189189 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a93599-a966-4dbe-b70f-1bf32b8a33e9" path="/var/lib/kubelet/pods/03a93599-a966-4dbe-b70f-1bf32b8a33e9/volumes" Sep 15 11:58:07.189671 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:07.189578 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a5fcc9-b549-45c2-9fe6-820968b8b614" path="/var/lib/kubelet/pods/30a5fcc9-b549-45c2-9fe6-820968b8b614/volumes" Sep 15 11:58:14.541454 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.541406 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb"] Sep 15 11:58:14.541956 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.541901 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a93599-a966-4dbe-b70f-1bf32b8a33e9" containerName="payload-pre-processing" Sep 15 11:58:14.541956 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.541921 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a93599-a966-4dbe-b70f-1bf32b8a33e9" containerName="payload-pre-processing" Sep 15 11:58:14.541956 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.541942 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a5fcc9-b549-45c2-9fe6-820968b8b614" containerName="payload-processing" Sep 15 11:58:14.541956 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.541951 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a5fcc9-b549-45c2-9fe6-820968b8b614" containerName="payload-processing" Sep 15 11:58:14.542186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.542001 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df23bdf4-9329-4cae-a665-e81bdf2b38be" containerName="maas-api" Sep 15 11:58:14.542186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.542012 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="df23bdf4-9329-4cae-a665-e81bdf2b38be" containerName="maas-api" Sep 15 11:58:14.542186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.542032 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6aeb71-7554-4b1f-b513-94481d491812" containerName="payload-processing" Sep 15 11:58:14.542186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.542043 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6aeb71-7554-4b1f-b513-94481d491812" containerName="payload-processing" Sep 15 11:58:14.542186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.542135 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a5fcc9-b549-45c2-9fe6-820968b8b614" containerName="payload-processing" Sep 15 11:58:14.542186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.542148 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="df23bdf4-9329-4cae-a665-e81bdf2b38be" containerName="maas-api" Sep 15 11:58:14.542186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.542157 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a93599-a966-4dbe-b70f-1bf32b8a33e9" containerName="payload-pre-processing" Sep 15 11:58:14.542186 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.542164 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6aeb71-7554-4b1f-b513-94481d491812" containerName="payload-processing" Sep 15 11:58:14.546665 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.546649 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.549808 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.549779 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"kube-root-ca.crt\"" Sep 15 11:58:14.549808 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.549804 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"openshift-service-ca.crt\"" Sep 15 11:58:14.549998 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.549781 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"e2e-conflict-a-6fb44257-kserve-self-signed-certs\"" Sep 15 11:58:14.549998 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.549933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ac76e5\"/\"default-dockercfg-mtvf9\"" Sep 15 11:58:14.554708 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.554688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb"] Sep 15 11:58:14.657494 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.657460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-dshm\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.657494 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.657496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-home\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.657706 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.657521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tmp-dir\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.657706 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.657606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-model-cache\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.657706 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.657630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tls-certs\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.657706 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.657667 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k86hf\" (UniqueName: \"kubernetes.io/projected/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-kube-api-access-k86hf\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.758959 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.758920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-dshm\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.758959 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.758954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-home\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.759241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.758980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tmp-dir\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.759241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.759010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-model-cache\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.759241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.759026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tls-certs\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.759241 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.759054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k86hf\" (UniqueName: \"kubernetes.io/projected/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-kube-api-access-k86hf\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.759476 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.759450 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-home\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.759528 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.759476 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-model-cache\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.759528 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.759500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tmp-dir\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.761338 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.761318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-dshm\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.761602 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.761583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tls-certs\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.768713 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.768695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k86hf\" (UniqueName: \"kubernetes.io/projected/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-kube-api-access-k86hf\") pod \"e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:14.856546 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:14.856467 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:15.133148 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:15.133119 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb"] Sep 15 11:58:15.135410 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:58:15.135375 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1c7cd2d_aef8_4bb3_b3a4_e2dcaded56e8.slice/crio-41fac9554f2efe45e5aac988ad6d8ad5cf3a6d4a2e78d39c560dfdc175fd3668 WatchSource:0}: Error finding container 41fac9554f2efe45e5aac988ad6d8ad5cf3a6d4a2e78d39c560dfdc175fd3668: Status 404 returned error can't find the container with id 41fac9554f2efe45e5aac988ad6d8ad5cf3a6d4a2e78d39c560dfdc175fd3668 Sep 15 11:58:15.606947 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:15.606910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" event={"ID":"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8","Type":"ContainerStarted","Data":"1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc"} Sep 15 11:58:15.607459 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:15.606960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" event={"ID":"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8","Type":"ContainerStarted","Data":"41fac9554f2efe45e5aac988ad6d8ad5cf3a6d4a2e78d39c560dfdc175fd3668"} Sep 15 11:58:15.625463 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:15.625394 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" podStartSLOduration=1.6253731679999999 podStartE2EDuration="1.625373168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:58:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:58:15.623909974 +0000 UTC m=+1182.969018593" watchObservedRunningTime="2026-09-15 11:58:15.625373168 +0000 UTC m=+1182.970481775" Sep 15 11:58:24.856788 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:24.856747 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:24.857294 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:24.857033 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:24.870554 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:24.870531 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:25.674347 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:25.674313 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 11:58:33.177441 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.177411 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:58:33.177895 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.177862 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 11:58:33.595065 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.594986 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w"] Sep 15 11:58:33.610080 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.610046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w"] Sep 15 11:58:33.610262 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.610185 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.612556 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.612524 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-ec8230-serving-cert\"" Sep 15 11:58:33.625173 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.625134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.625173 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.625183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.625412 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.625251 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw886\" (UniqueName: \"kubernetes.io/projected/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-kube-api-access-lw886\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.625412 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.625280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.726448 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.726411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.726650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.726457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.726650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.726492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw886\" (UniqueName: \"kubernetes.io/projected/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-kube-api-access-lw886\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.726650 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.726511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.727071 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.727050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.729176 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.729152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.729301 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.729191 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.735194 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.735176 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw886\" (UniqueName: \"kubernetes.io/projected/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-kube-api-access-lw886\") pod \"maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:33.924095 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:33.924014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:34.206824 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.206795 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w"] Sep 15 11:58:34.207812 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:58:34.207784 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cb7a8d_b7ab_48c3_b4b4_e420d50536c8.slice/crio-6009da2b2092a80bf20872997ba85f3728f55ea5008e01cc4304eb26fc66f1a3 WatchSource:0}: Error finding container 6009da2b2092a80bf20872997ba85f3728f55ea5008e01cc4304eb26fc66f1a3: Status 404 returned error can't find the container with id 6009da2b2092a80bf20872997ba85f3728f55ea5008e01cc4304eb26fc66f1a3 Sep 15 11:58:34.280447 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.280409 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq"] Sep 15 11:58:34.285360 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.285343 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:34.287736 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.287713 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-ec8230\"" Sep 15 11:58:34.287736 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.287716 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-ec8230-dockercfg-p5tpb\"" Sep 15 11:58:34.294513 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.294492 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq"] Sep 15 11:58:34.332782 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.332753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpdtg\" (UniqueName: \"kubernetes.io/projected/da0b566f-95d3-4f6d-8a8c-339e356a3013-kube-api-access-hpdtg\") pod \"payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq\" (UID: \"da0b566f-95d3-4f6d-8a8c-339e356a3013\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:34.332952 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.332840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b566f-95d3-4f6d-8a8c-339e356a3013-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq\" (UID: \"da0b566f-95d3-4f6d-8a8c-339e356a3013\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:34.433669 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.433622 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b566f-95d3-4f6d-8a8c-339e356a3013-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq\" (UID: \"da0b566f-95d3-4f6d-8a8c-339e356a3013\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:34.433852 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.433740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpdtg\" (UniqueName: \"kubernetes.io/projected/da0b566f-95d3-4f6d-8a8c-339e356a3013-kube-api-access-hpdtg\") pod \"payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq\" (UID: \"da0b566f-95d3-4f6d-8a8c-339e356a3013\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:34.434400 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.434380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b566f-95d3-4f6d-8a8c-339e356a3013-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq\" (UID: \"da0b566f-95d3-4f6d-8a8c-339e356a3013\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:34.444480 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.444453 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpdtg\" (UniqueName: \"kubernetes.io/projected/da0b566f-95d3-4f6d-8a8c-339e356a3013-kube-api-access-hpdtg\") pod \"payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq\" (UID: \"da0b566f-95d3-4f6d-8a8c-339e356a3013\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:34.597861 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.597738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:34.714053 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.714019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" event={"ID":"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8","Type":"ContainerStarted","Data":"6009da2b2092a80bf20872997ba85f3728f55ea5008e01cc4304eb26fc66f1a3"} Sep 15 11:58:34.875725 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:34.875631 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq"] Sep 15 11:58:34.882621 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:58:34.882584 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0b566f_95d3_4f6d_8a8c_339e356a3013.slice/crio-fc1a5a4d99148d9fd87a3693424aff9b11ef8605ae35f414f836f6e27cfb348a WatchSource:0}: Error finding container fc1a5a4d99148d9fd87a3693424aff9b11ef8605ae35f414f836f6e27cfb348a: Status 404 returned error can't find the container with id fc1a5a4d99148d9fd87a3693424aff9b11ef8605ae35f414f836f6e27cfb348a Sep 15 11:58:35.722066 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:35.722026 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" event={"ID":"da0b566f-95d3-4f6d-8a8c-339e356a3013","Type":"ContainerStarted","Data":"40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735"} Sep 15 11:58:35.722546 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:35.722092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" event={"ID":"da0b566f-95d3-4f6d-8a8c-339e356a3013","Type":"ContainerStarted","Data":"fc1a5a4d99148d9fd87a3693424aff9b11ef8605ae35f414f836f6e27cfb348a"} Sep 15 11:58:35.722546 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:35.722123 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:58:35.723840 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:35.723811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" event={"ID":"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8","Type":"ContainerStarted","Data":"badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8"} Sep 15 11:58:35.723987 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:35.723971 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:35.742717 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:35.742667 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" podStartSLOduration=1.742653196 podStartE2EDuration="1.742653196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:58:35.740472907 +0000 UTC m=+1203.085581514" watchObservedRunningTime="2026-09-15 11:58:35.742653196 +0000 UTC m=+1203.087761801" Sep 15 11:58:35.756614 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:35.756550 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" podStartSLOduration=2.28384298 podStartE2EDuration="2.75653388s" totalImagesPullingTime="472.6909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:58:35.755775019 +0000 UTC m=+1203.100883625" watchObservedRunningTime="2026-09-15 11:58:35.75653388 +0000 UTC m=+1203.101642487" Sep 15 11:58:41.736600 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:41.736571 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 11:58:46.731801 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:58:46.731773 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 11:59:08.775159 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.775126 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6"] Sep 15 11:59:08.779246 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.779225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.781399 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.781375 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-6dca39-serving-cert\"" Sep 15 11:59:08.790744 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.790721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6"] Sep 15 11:59:08.863742 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.863709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.863742 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.863750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fnv\" (UniqueName: \"kubernetes.io/projected/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-kube-api-access-42fnv\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.863963 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.863857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.863963 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.863918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.965086 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.965054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.965086 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.965086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fnv\" (UniqueName: \"kubernetes.io/projected/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-kube-api-access-42fnv\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.965348 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.965129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.965348 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.965167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.965886 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.965821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.967600 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.967581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.967791 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.967773 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:08.976310 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:08.976289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fnv\" (UniqueName: \"kubernetes.io/projected/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-kube-api-access-42fnv\") pod \"maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:09.090351 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.090267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:09.365399 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.365367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6"] Sep 15 11:59:09.368391 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:59:09.368356 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2797a9_934a_4be7_9fd5_76faf0cf3b35.slice/crio-d2d1c2d9589af0e0cb70d23d52f12b40a757ed4f4560e8727892ae1be9f39b8b WatchSource:0}: Error finding container d2d1c2d9589af0e0cb70d23d52f12b40a757ed4f4560e8727892ae1be9f39b8b: Status 404 returned error can't find the container with id d2d1c2d9589af0e0cb70d23d52f12b40a757ed4f4560e8727892ae1be9f39b8b Sep 15 11:59:09.479822 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.479784 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch"] Sep 15 11:59:09.484912 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.484889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:09.487193 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.487173 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6dca39\"" Sep 15 11:59:09.487314 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.487174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6dca39-dockercfg-4bl8b\"" Sep 15 11:59:09.494437 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.494415 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch"] Sep 15 11:59:09.571400 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.571366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9r7r\" (UniqueName: \"kubernetes.io/projected/64fbe269-8c48-4364-9721-6a5db61f69ab-kube-api-access-g9r7r\") pod \"payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch\" (UID: \"64fbe269-8c48-4364-9721-6a5db61f69ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:09.571605 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.571440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbe269-8c48-4364-9721-6a5db61f69ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch\" (UID: \"64fbe269-8c48-4364-9721-6a5db61f69ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:09.672471 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.672379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9r7r\" (UniqueName: \"kubernetes.io/projected/64fbe269-8c48-4364-9721-6a5db61f69ab-kube-api-access-g9r7r\") pod \"payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch\" (UID: \"64fbe269-8c48-4364-9721-6a5db61f69ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:09.672471 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.672449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbe269-8c48-4364-9721-6a5db61f69ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch\" (UID: \"64fbe269-8c48-4364-9721-6a5db61f69ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:09.673134 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.673113 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbe269-8c48-4364-9721-6a5db61f69ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch\" (UID: \"64fbe269-8c48-4364-9721-6a5db61f69ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:09.684017 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.683995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9r7r\" (UniqueName: \"kubernetes.io/projected/64fbe269-8c48-4364-9721-6a5db61f69ab-kube-api-access-g9r7r\") pod \"payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch\" (UID: \"64fbe269-8c48-4364-9721-6a5db61f69ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:09.797693 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.797661 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:09.913271 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:09.913238 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" event={"ID":"6d2797a9-934a-4be7-9fd5-76faf0cf3b35","Type":"ContainerStarted","Data":"d2d1c2d9589af0e0cb70d23d52f12b40a757ed4f4560e8727892ae1be9f39b8b"} Sep 15 11:59:10.074511 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:10.074483 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch"] Sep 15 11:59:10.076347 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:59:10.076318 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fbe269_8c48_4364_9721_6a5db61f69ab.slice/crio-b699387ee80a3ba0740001fe1ab9d39b3c19224691b62aadd35570eb56cfbdec WatchSource:0}: Error finding container b699387ee80a3ba0740001fe1ab9d39b3c19224691b62aadd35570eb56cfbdec: Status 404 returned error can't find the container with id b699387ee80a3ba0740001fe1ab9d39b3c19224691b62aadd35570eb56cfbdec Sep 15 11:59:10.920290 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:10.920251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" event={"ID":"64fbe269-8c48-4364-9721-6a5db61f69ab","Type":"ContainerStarted","Data":"8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998"} Sep 15 11:59:10.920727 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:10.920302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" event={"ID":"64fbe269-8c48-4364-9721-6a5db61f69ab","Type":"ContainerStarted","Data":"b699387ee80a3ba0740001fe1ab9d39b3c19224691b62aadd35570eb56cfbdec"} Sep 15 11:59:10.920727 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:10.920403 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:10.939833 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:10.939785 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" podStartSLOduration=1.9397698079999999 podStartE2EDuration="1.939769808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:10.938524434 +0000 UTC m=+1238.283633038" watchObservedRunningTime="2026-09-15 11:59:10.939769808 +0000 UTC m=+1238.284878413" Sep 15 11:59:11.927068 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:11.927030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" event={"ID":"6d2797a9-934a-4be7-9fd5-76faf0cf3b35","Type":"ContainerStarted","Data":"5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2"} Sep 15 11:59:11.927484 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:11.927125 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:11.946536 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:11.946491 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" podStartSLOduration=2.091814172 podStartE2EDuration="3.946475961s" totalImagesPullingTime="1.854661789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:11.945658685 +0000 UTC m=+1239.290767291" watchObservedRunningTime="2026-09-15 11:59:11.946475961 +0000 UTC m=+1239.291584610" Sep 15 11:59:17.939293 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:17.939262 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 11:59:21.930024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:21.929987 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 11:59:34.799289 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:34.799249 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7878-339f6a"] Sep 15 11:59:34.802045 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:34.802026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-339f6a" Sep 15 11:59:34.804568 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:34.804548 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6twr\"" Sep 15 11:59:34.811006 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:34.810458 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7878-339f6a"] Sep 15 11:59:34.910167 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:34.910129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvwdr\" (UniqueName: \"kubernetes.io/projected/8d2845e2-42e1-4b63-9031-5f77b39be0a3-kube-api-access-wvwdr\") pod \"test-curl-7878-339f6a\" (UID: \"8d2845e2-42e1-4b63-9031-5f77b39be0a3\") " pod="opendatahub/test-curl-7878-339f6a" Sep 15 11:59:35.011589 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:35.011558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvwdr\" (UniqueName: \"kubernetes.io/projected/8d2845e2-42e1-4b63-9031-5f77b39be0a3-kube-api-access-wvwdr\") pod \"test-curl-7878-339f6a\" (UID: \"8d2845e2-42e1-4b63-9031-5f77b39be0a3\") " pod="opendatahub/test-curl-7878-339f6a" Sep 15 11:59:35.020626 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:35.020604 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvwdr\" (UniqueName: \"kubernetes.io/projected/8d2845e2-42e1-4b63-9031-5f77b39be0a3-kube-api-access-wvwdr\") pod \"test-curl-7878-339f6a\" (UID: \"8d2845e2-42e1-4b63-9031-5f77b39be0a3\") " pod="opendatahub/test-curl-7878-339f6a" Sep 15 11:59:35.115575 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:35.115498 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-339f6a" Sep 15 11:59:35.387814 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:35.387717 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7878-339f6a"] Sep 15 11:59:35.390116 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:59:35.390088 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d2845e2_42e1_4b63_9031_5f77b39be0a3.slice/crio-c60f06f7774f7a2e3ad6ee93e329426786acc457bd71a5a47aeedcfbdd794e82 WatchSource:0}: Error finding container c60f06f7774f7a2e3ad6ee93e329426786acc457bd71a5a47aeedcfbdd794e82: Status 404 returned error can't find the container with id c60f06f7774f7a2e3ad6ee93e329426786acc457bd71a5a47aeedcfbdd794e82 Sep 15 11:59:36.063968 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.063927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7878-339f6a" event={"ID":"8d2845e2-42e1-4b63-9031-5f77b39be0a3","Type":"ContainerStarted","Data":"aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e"} Sep 15 11:59:36.063968 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.063970 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7878-339f6a" event={"ID":"8d2845e2-42e1-4b63-9031-5f77b39be0a3","Type":"ContainerStarted","Data":"c60f06f7774f7a2e3ad6ee93e329426786acc457bd71a5a47aeedcfbdd794e82"} Sep 15 11:59:36.082300 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.082252 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7878-339f6a" podStartSLOduration=2.082237397 podStartE2EDuration="2.082237397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:36.080449321 +0000 UTC m=+1263.425557929" watchObservedRunningTime="2026-09-15 11:59:36.082237397 +0000 UTC m=+1263.427346026" Sep 15 11:59:36.668669 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.668632 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7878-339f6a"] Sep 15 11:59:36.673432 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.673402 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7878-339f6a"] Sep 15 11:59:36.792591 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.792508 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7878-7abecb"] Sep 15 11:59:36.793033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.793018 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2845e2-42e1-4b63-9031-5f77b39be0a3" containerName="test-curl-7878-339f6a" Sep 15 11:59:36.793075 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.793035 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2845e2-42e1-4b63-9031-5f77b39be0a3" containerName="test-curl-7878-339f6a" Sep 15 11:59:36.793115 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.793105 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2845e2-42e1-4b63-9031-5f77b39be0a3" containerName="test-curl-7878-339f6a" Sep 15 11:59:36.795989 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.795970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:36.803333 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.803307 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7878-7abecb"] Sep 15 11:59:36.929791 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:36.929756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pmx6\" (UniqueName: \"kubernetes.io/projected/b60d631a-fc21-4fe1-a691-4f52eba7477d-kube-api-access-6pmx6\") pod \"test-curl-7878-7abecb\" (UID: \"b60d631a-fc21-4fe1-a691-4f52eba7477d\") " pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:37.030847 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:37.030809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pmx6\" (UniqueName: \"kubernetes.io/projected/b60d631a-fc21-4fe1-a691-4f52eba7477d-kube-api-access-6pmx6\") pod \"test-curl-7878-7abecb\" (UID: \"b60d631a-fc21-4fe1-a691-4f52eba7477d\") " pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:37.040791 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:37.040758 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pmx6\" (UniqueName: \"kubernetes.io/projected/b60d631a-fc21-4fe1-a691-4f52eba7477d-kube-api-access-6pmx6\") pod \"test-curl-7878-7abecb\" (UID: \"b60d631a-fc21-4fe1-a691-4f52eba7477d\") " pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:37.069230 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:37.069135 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7878-339f6a" container="test-curl-7878-339f6a" Sep 15 11:59:37.108077 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:37.108046 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:37.381224 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:37.381187 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7878-7abecb"] Sep 15 11:59:37.383131 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:59:37.383099 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb60d631a_fc21_4fe1_a691_4f52eba7477d.slice/crio-9e988d9233df6b908d76e9293f4d7b26ea1189728633459d9348997b057ef993 WatchSource:0}: Error finding container 9e988d9233df6b908d76e9293f4d7b26ea1189728633459d9348997b057ef993: Status 404 returned error can't find the container with id 9e988d9233df6b908d76e9293f4d7b26ea1189728633459d9348997b057ef993 Sep 15 11:59:38.076731 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.076693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7878-7abecb" event={"ID":"b60d631a-fc21-4fe1-a691-4f52eba7477d","Type":"ContainerStarted","Data":"4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719"} Sep 15 11:59:38.077120 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.076745 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7878-7abecb" event={"ID":"b60d631a-fc21-4fe1-a691-4f52eba7477d","Type":"ContainerStarted","Data":"9e988d9233df6b908d76e9293f4d7b26ea1189728633459d9348997b057ef993"} Sep 15 11:59:38.077120 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.076842 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7878-339f6a" podUID="8d2845e2-42e1-4b63-9031-5f77b39be0a3" containerName="test-curl-7878-339f6a" containerID="cri-o://aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e" gracePeriod=2 Sep 15 11:59:38.092505 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.092458 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7878-7abecb" podStartSLOduration=2.092440585 podStartE2EDuration="2.092440585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:38.091583538 +0000 UTC m=+1265.436692161" watchObservedRunningTime="2026-09-15 11:59:38.092440585 +0000 UTC m=+1265.437549241" Sep 15 11:59:38.683295 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.683263 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7878-7abecb"] Sep 15 11:59:38.692727 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.692700 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7878-7abecb"] Sep 15 11:59:38.806562 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.806470 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7878-54b40d"] Sep 15 11:59:38.807124 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.807109 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b60d631a-fc21-4fe1-a691-4f52eba7477d" containerName="test-curl-7878-7abecb" Sep 15 11:59:38.807187 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.807128 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b60d631a-fc21-4fe1-a691-4f52eba7477d" containerName="test-curl-7878-7abecb" Sep 15 11:59:38.807284 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.807260 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b60d631a-fc21-4fe1-a691-4f52eba7477d" containerName="test-curl-7878-7abecb" Sep 15 11:59:38.810405 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.810388 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-54b40d" Sep 15 11:59:38.814632 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.814610 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7878-54b40d"] Sep 15 11:59:38.952290 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:38.952243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shch\" (UniqueName: \"kubernetes.io/projected/180ae443-805d-44c0-b7c9-4b7f37526042-kube-api-access-7shch\") pod \"test-curl-7878-54b40d\" (UID: \"180ae443-805d-44c0-b7c9-4b7f37526042\") " pod="opendatahub/test-curl-7878-54b40d" Sep 15 11:59:39.053361 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:39.053323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shch\" (UniqueName: \"kubernetes.io/projected/180ae443-805d-44c0-b7c9-4b7f37526042-kube-api-access-7shch\") pod \"test-curl-7878-54b40d\" (UID: \"180ae443-805d-44c0-b7c9-4b7f37526042\") " pod="opendatahub/test-curl-7878-54b40d" Sep 15 11:59:39.062300 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:39.062230 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shch\" (UniqueName: \"kubernetes.io/projected/180ae443-805d-44c0-b7c9-4b7f37526042-kube-api-access-7shch\") pod \"test-curl-7878-54b40d\" (UID: \"180ae443-805d-44c0-b7c9-4b7f37526042\") " pod="opendatahub/test-curl-7878-54b40d" Sep 15 11:59:39.081849 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:39.081822 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7878-7abecb" container="test-curl-7878-7abecb" Sep 15 11:59:39.123041 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:39.123013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-54b40d" Sep 15 11:59:39.402018 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:39.401806 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7878-54b40d"] Sep 15 11:59:39.405679 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:59:39.405637 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod180ae443_805d_44c0_b7c9_4b7f37526042.slice/crio-471a96b775883e7e8d151586151befe1f780d76f5d3b27327b5d5c539db528d7 WatchSource:0}: Error finding container 471a96b775883e7e8d151586151befe1f780d76f5d3b27327b5d5c539db528d7: Status 404 returned error can't find the container with id 471a96b775883e7e8d151586151befe1f780d76f5d3b27327b5d5c539db528d7 Sep 15 11:59:40.090542 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.090506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7878-54b40d" event={"ID":"180ae443-805d-44c0-b7c9-4b7f37526042","Type":"ContainerStarted","Data":"d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc"} Sep 15 11:59:40.090955 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.090559 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7878-54b40d" event={"ID":"180ae443-805d-44c0-b7c9-4b7f37526042","Type":"ContainerStarted","Data":"471a96b775883e7e8d151586151befe1f780d76f5d3b27327b5d5c539db528d7"} Sep 15 11:59:40.090955 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.090618 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7878-7abecb" podUID="b60d631a-fc21-4fe1-a691-4f52eba7477d" containerName="test-curl-7878-7abecb" containerID="cri-o://4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719" gracePeriod=2 Sep 15 11:59:40.108628 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.108576 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7878-54b40d" podStartSLOduration=2.108557184 podStartE2EDuration="2.108557184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:40.108376878 +0000 UTC m=+1267.453485486" watchObservedRunningTime="2026-09-15 11:59:40.108557184 +0000 UTC m=+1267.453665791" Sep 15 11:59:40.227886 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.227862 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-339f6a" Sep 15 11:59:40.363797 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.363719 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvwdr\" (UniqueName: \"kubernetes.io/projected/8d2845e2-42e1-4b63-9031-5f77b39be0a3-kube-api-access-wvwdr\") pod \"8d2845e2-42e1-4b63-9031-5f77b39be0a3\" (UID: \"8d2845e2-42e1-4b63-9031-5f77b39be0a3\") " Sep 15 11:59:40.365861 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.365828 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2845e2-42e1-4b63-9031-5f77b39be0a3-kube-api-access-wvwdr" (OuterVolumeSpecName: "kube-api-access-wvwdr") pod "8d2845e2-42e1-4b63-9031-5f77b39be0a3" (UID: "8d2845e2-42e1-4b63-9031-5f77b39be0a3"). InnerVolumeSpecName "kube-api-access-wvwdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:59:40.465334 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.465306 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvwdr\" (UniqueName: \"kubernetes.io/projected/8d2845e2-42e1-4b63-9031-5f77b39be0a3-kube-api-access-wvwdr\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:59:40.685772 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.685733 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7878-54b40d"] Sep 15 11:59:40.693730 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.693699 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7878-54b40d"] Sep 15 11:59:40.804467 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.804425 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7878-256646"] Sep 15 11:59:40.804935 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.804920 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="180ae443-805d-44c0-b7c9-4b7f37526042" containerName="test-curl-7878-54b40d" Sep 15 11:59:40.804979 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.804940 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="180ae443-805d-44c0-b7c9-4b7f37526042" containerName="test-curl-7878-54b40d" Sep 15 11:59:40.805033 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.805024 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="180ae443-805d-44c0-b7c9-4b7f37526042" containerName="test-curl-7878-54b40d" Sep 15 11:59:40.808300 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.808284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-256646" Sep 15 11:59:40.814802 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.814775 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7878-256646"] Sep 15 11:59:40.969412 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:40.969330 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nl5\" (UniqueName: \"kubernetes.io/projected/bfecf7fd-7145-4375-b904-1d876c2b8014-kube-api-access-q9nl5\") pod \"test-curl-7878-256646\" (UID: \"bfecf7fd-7145-4375-b904-1d876c2b8014\") " pod="opendatahub/test-curl-7878-256646" Sep 15 11:59:41.070422 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.070384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nl5\" (UniqueName: \"kubernetes.io/projected/bfecf7fd-7145-4375-b904-1d876c2b8014-kube-api-access-q9nl5\") pod \"test-curl-7878-256646\" (UID: \"bfecf7fd-7145-4375-b904-1d876c2b8014\") " pod="opendatahub/test-curl-7878-256646" Sep 15 11:59:41.080111 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.080068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nl5\" (UniqueName: \"kubernetes.io/projected/bfecf7fd-7145-4375-b904-1d876c2b8014-kube-api-access-q9nl5\") pod \"test-curl-7878-256646\" (UID: \"bfecf7fd-7145-4375-b904-1d876c2b8014\") " pod="opendatahub/test-curl-7878-256646" Sep 15 11:59:41.096700 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.096673 2579 generic.go:356] "Generic (PLEG): container finished" podID="8d2845e2-42e1-4b63-9031-5f77b39be0a3" containerID="aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e" exitCode=137 Sep 15 11:59:41.097078 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.096721 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-339f6a" Sep 15 11:59:41.097078 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.096745 2579 scope.go:117] "RemoveContainer" containerID="aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e" Sep 15 11:59:41.097078 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:41.096972 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7878-54b40d" container="test-curl-7878-54b40d" Sep 15 11:59:41.107008 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.106988 2579 scope.go:117] "RemoveContainer" containerID="aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e" Sep 15 11:59:41.107392 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:41.107367 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e\": container with ID starting with aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e not found: ID does not exist" containerID="aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e" Sep 15 11:59:41.107469 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.107404 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e"} err="failed to get container status \"aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e\": rpc error: code = NotFound desc = could not find container \"aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e\": container with ID starting with aff58df262c63cdfcd32960102493d2e9df4d12c32fd0f7159c962ae5f39fa9e not found: ID does not exist" Sep 15 11:59:41.110096 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:41.110074 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7878-339f6a\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="8d2845e2-42e1-4b63-9031-5f77b39be0a3" pod="opendatahub/test-curl-7878-339f6a" Sep 15 11:59:41.119801 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.119780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-256646" Sep 15 11:59:41.190772 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.190733 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d2845e2-42e1-4b63-9031-5f77b39be0a3" path="/var/lib/kubelet/pods/8d2845e2-42e1-4b63-9031-5f77b39be0a3/volumes" Sep 15 11:59:41.393024 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:41.392937 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7878-256646"] Sep 15 11:59:41.396043 ip-10-0-140-16 kubenswrapper[2579]: W0915 11:59:41.396005 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfecf7fd_7145_4375_b904_1d876c2b8014.slice/crio-f3fb55f5ba1d80567de024efd89cc299e2bea8bcdc16f85905d52ac154eea2fd WatchSource:0}: Error finding container f3fb55f5ba1d80567de024efd89cc299e2bea8bcdc16f85905d52ac154eea2fd: Status 404 returned error can't find the container with id f3fb55f5ba1d80567de024efd89cc299e2bea8bcdc16f85905d52ac154eea2fd Sep 15 11:59:42.106360 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.106329 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7878-256646" event={"ID":"bfecf7fd-7145-4375-b904-1d876c2b8014","Type":"ContainerStarted","Data":"5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab"} Sep 15 11:59:42.106758 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.106377 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7878-256646" event={"ID":"bfecf7fd-7145-4375-b904-1d876c2b8014","Type":"ContainerStarted","Data":"f3fb55f5ba1d80567de024efd89cc299e2bea8bcdc16f85905d52ac154eea2fd"} Sep 15 11:59:42.106758 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.106404 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7878-54b40d" podUID="180ae443-805d-44c0-b7c9-4b7f37526042" containerName="test-curl-7878-54b40d" containerID="cri-o://d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc" gracePeriod=2 Sep 15 11:59:42.121013 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.120960 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7878-256646" podStartSLOduration=2.120941042 podStartE2EDuration="2.120941042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:59:42.120765172 +0000 UTC m=+1269.465873791" watchObservedRunningTime="2026-09-15 11:59:42.120941042 +0000 UTC m=+1269.466049648" Sep 15 11:59:42.283915 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.283892 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:42.383462 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.383384 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pmx6\" (UniqueName: \"kubernetes.io/projected/b60d631a-fc21-4fe1-a691-4f52eba7477d-kube-api-access-6pmx6\") pod \"b60d631a-fc21-4fe1-a691-4f52eba7477d\" (UID: \"b60d631a-fc21-4fe1-a691-4f52eba7477d\") " Sep 15 11:59:42.385468 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.385447 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b60d631a-fc21-4fe1-a691-4f52eba7477d-kube-api-access-6pmx6" (OuterVolumeSpecName: "kube-api-access-6pmx6") pod "b60d631a-fc21-4fe1-a691-4f52eba7477d" (UID: "b60d631a-fc21-4fe1-a691-4f52eba7477d"). InnerVolumeSpecName "kube-api-access-6pmx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:59:42.484009 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.483975 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pmx6\" (UniqueName: \"kubernetes.io/projected/b60d631a-fc21-4fe1-a691-4f52eba7477d-kube-api-access-6pmx6\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:59:42.704749 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.704709 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7878-256646"] Sep 15 11:59:42.709077 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:42.709037 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7878-256646"] Sep 15 11:59:43.113231 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:43.113125 2579 generic.go:356] "Generic (PLEG): container finished" podID="b60d631a-fc21-4fe1-a691-4f52eba7477d" containerID="4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719" exitCode=137 Sep 15 11:59:43.113231 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:43.113170 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:43.113726 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:43.113257 2579 scope.go:117] "RemoveContainer" containerID="4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719" Sep 15 11:59:43.113726 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:43.113484 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7878-256646" container="test-curl-7878-256646" Sep 15 11:59:43.122673 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:43.122657 2579 scope.go:117] "RemoveContainer" containerID="4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719" Sep 15 11:59:43.122964 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:43.122947 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719\": container with ID starting with 4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719 not found: ID does not exist" containerID="4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719" Sep 15 11:59:43.123017 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:43.122972 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719"} err="failed to get container status \"4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719\": rpc error: code = NotFound desc = could not find container \"4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719\": container with ID starting with 4dafcdb164eec677e193537ecb0d4907ce83c294ad2d142a67521f1b8fb52719 not found: ID does not exist" Sep 15 11:59:43.123602 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:43.123573 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7878-7abecb\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="b60d631a-fc21-4fe1-a691-4f52eba7477d" pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:43.186942 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:43.186910 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7878-7abecb\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="b60d631a-fc21-4fe1-a691-4f52eba7477d" pod="opendatahub/test-curl-7878-7abecb" Sep 15 11:59:43.189135 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:43.189110 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b60d631a-fc21-4fe1-a691-4f52eba7477d" path="/var/lib/kubelet/pods/b60d631a-fc21-4fe1-a691-4f52eba7477d/volumes" Sep 15 11:59:44.126234 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:44.119871 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7878-256646" podUID="bfecf7fd-7145-4375-b904-1d876c2b8014" containerName="test-curl-7878-256646" containerID="cri-o://5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab" gracePeriod=2 Sep 15 11:59:44.303105 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:44.303075 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-54b40d" Sep 15 11:59:44.403112 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:44.403027 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7shch\" (UniqueName: \"kubernetes.io/projected/180ae443-805d-44c0-b7c9-4b7f37526042-kube-api-access-7shch\") pod \"180ae443-805d-44c0-b7c9-4b7f37526042\" (UID: \"180ae443-805d-44c0-b7c9-4b7f37526042\") " Sep 15 11:59:44.405426 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:44.405391 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/180ae443-805d-44c0-b7c9-4b7f37526042-kube-api-access-7shch" (OuterVolumeSpecName: "kube-api-access-7shch") pod "180ae443-805d-44c0-b7c9-4b7f37526042" (UID: "180ae443-805d-44c0-b7c9-4b7f37526042"). InnerVolumeSpecName "kube-api-access-7shch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:59:44.504481 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:44.504440 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7shch\" (UniqueName: \"kubernetes.io/projected/180ae443-805d-44c0-b7c9-4b7f37526042-kube-api-access-7shch\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:59:45.126066 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:45.126025 2579 generic.go:356] "Generic (PLEG): container finished" podID="180ae443-805d-44c0-b7c9-4b7f37526042" containerID="d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc" exitCode=137 Sep 15 11:59:45.126066 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:45.126067 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-54b40d" Sep 15 11:59:45.126321 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:45.126092 2579 scope.go:117] "RemoveContainer" containerID="d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc" Sep 15 11:59:45.135411 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:45.135228 2579 scope.go:117] "RemoveContainer" containerID="d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc" Sep 15 11:59:45.135532 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:45.135511 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc\": container with ID starting with d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc not found: ID does not exist" containerID="d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc" Sep 15 11:59:45.135594 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:45.135540 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc"} err="failed to get container status \"d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc\": rpc error: code = NotFound desc = could not find container \"d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc\": container with ID starting with d352ba060dfd06d908c9a63117b86c92181c3d7485d83bd101cc31753c682fbc not found: ID does not exist" Sep 15 11:59:45.136587 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:45.136562 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7878-54b40d\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="180ae443-805d-44c0-b7c9-4b7f37526042" pod="opendatahub/test-curl-7878-54b40d" Sep 15 11:59:45.189884 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:45.189837 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="180ae443-805d-44c0-b7c9-4b7f37526042" path="/var/lib/kubelet/pods/180ae443-805d-44c0-b7c9-4b7f37526042/volumes" Sep 15 11:59:46.328353 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:46.328331 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-256646" Sep 15 11:59:46.423617 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:46.423521 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9nl5\" (UniqueName: \"kubernetes.io/projected/bfecf7fd-7145-4375-b904-1d876c2b8014-kube-api-access-q9nl5\") pod \"bfecf7fd-7145-4375-b904-1d876c2b8014\" (UID: \"bfecf7fd-7145-4375-b904-1d876c2b8014\") " Sep 15 11:59:46.425842 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:46.425803 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfecf7fd-7145-4375-b904-1d876c2b8014-kube-api-access-q9nl5" (OuterVolumeSpecName: "kube-api-access-q9nl5") pod "bfecf7fd-7145-4375-b904-1d876c2b8014" (UID: "bfecf7fd-7145-4375-b904-1d876c2b8014"). InnerVolumeSpecName "kube-api-access-q9nl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 11:59:46.524425 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:46.524397 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9nl5\" (UniqueName: \"kubernetes.io/projected/bfecf7fd-7145-4375-b904-1d876c2b8014-kube-api-access-q9nl5\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 11:59:47.139309 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:47.139274 2579 generic.go:356] "Generic (PLEG): container finished" podID="bfecf7fd-7145-4375-b904-1d876c2b8014" containerID="5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab" exitCode=137 Sep 15 11:59:47.139490 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:47.139315 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7878-256646" Sep 15 11:59:47.139490 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:47.139361 2579 scope.go:117] "RemoveContainer" containerID="5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab" Sep 15 11:59:47.148448 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:47.148422 2579 scope.go:117] "RemoveContainer" containerID="5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab" Sep 15 11:59:47.148758 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:47.148735 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab\": container with ID starting with 5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab not found: ID does not exist" containerID="5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab" Sep 15 11:59:47.148837 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:47.148767 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab"} err="failed to get container status \"5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab\": rpc error: code = NotFound desc = could not find container \"5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab\": container with ID starting with 5fdb033cc4269b7adfcf23e8a8ca64ef844b4eb78e7a93c5320b61f1eb44ebab not found: ID does not exist" Sep 15 11:59:47.149370 ip-10-0-140-16 kubenswrapper[2579]: E0915 11:59:47.149341 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7878-256646\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="bfecf7fd-7145-4375-b904-1d876c2b8014" pod="opendatahub/test-curl-7878-256646" Sep 15 11:59:47.188950 ip-10-0-140-16 kubenswrapper[2579]: I0915 11:59:47.188919 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfecf7fd-7145-4375-b904-1d876c2b8014" path="/var/lib/kubelet/pods/bfecf7fd-7145-4375-b904-1d876c2b8014/volumes" Sep 15 12:00:03.139504 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.139462 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9"] Sep 15 12:00:03.139917 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.139711 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" podUID="6e1e4751-4c95-4e92-bb96-1934ead2ffb4" containerName="maas-api" containerID="cri-o://dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd" gracePeriod=30 Sep 15 12:00:03.410988 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.410959 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 12:00:03.480679 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.480651 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-openshift-service-ca\") pod \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " Sep 15 12:00:03.480895 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.480745 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-metrics-certs\") pod \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " Sep 15 12:00:03.480895 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.480823 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7728\" (UniqueName: \"kubernetes.io/projected/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-kube-api-access-v7728\") pod \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " Sep 15 12:00:03.480895 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.480850 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-maas-api-tls\") pod \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\" (UID: \"6e1e4751-4c95-4e92-bb96-1934ead2ffb4\") " Sep 15 12:00:03.481087 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.481057 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e1e4751-4c95-4e92-bb96-1934ead2ffb4" (UID: "6e1e4751-4c95-4e92-bb96-1934ead2ffb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:00:03.481249 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.481231 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:00:03.483508 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.483478 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-kube-api-access-v7728" (OuterVolumeSpecName: "kube-api-access-v7728") pod "6e1e4751-4c95-4e92-bb96-1934ead2ffb4" (UID: "6e1e4751-4c95-4e92-bb96-1934ead2ffb4"). InnerVolumeSpecName "kube-api-access-v7728". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:00:03.483687 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.483663 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e1e4751-4c95-4e92-bb96-1934ead2ffb4" (UID: "6e1e4751-4c95-4e92-bb96-1934ead2ffb4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:00:03.483687 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.483676 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e1e4751-4c95-4e92-bb96-1934ead2ffb4" (UID: "6e1e4751-4c95-4e92-bb96-1934ead2ffb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:00:03.582584 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.582553 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7728\" (UniqueName: \"kubernetes.io/projected/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-kube-api-access-v7728\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:00:03.582584 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.582582 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:00:03.582584 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:03.582591 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e1e4751-4c95-4e92-bb96-1934ead2ffb4-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:00:04.246360 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.246326 2579 generic.go:356] "Generic (PLEG): container finished" podID="6e1e4751-4c95-4e92-bb96-1934ead2ffb4" containerID="dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd" exitCode=0 Sep 15 12:00:04.246738 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.246371 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" Sep 15 12:00:04.246738 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.246370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" event={"ID":"6e1e4751-4c95-4e92-bb96-1934ead2ffb4","Type":"ContainerDied","Data":"dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd"} Sep 15 12:00:04.246738 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.246503 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9" event={"ID":"6e1e4751-4c95-4e92-bb96-1934ead2ffb4","Type":"ContainerDied","Data":"9349ec78833551bdf815a5ba34942ac5d39662adc35fd9a368a279a9c350bf6e"} Sep 15 12:00:04.246738 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.246533 2579 scope.go:117] "RemoveContainer" containerID="dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd" Sep 15 12:00:04.258613 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.258594 2579 scope.go:117] "RemoveContainer" containerID="dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd" Sep 15 12:00:04.259141 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:00:04.259118 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd\": container with ID starting with dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd not found: ID does not exist" containerID="dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd" Sep 15 12:00:04.259277 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.259148 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd"} err="failed to get container status \"dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd\": rpc error: code = NotFound desc = could not find container \"dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd\": container with ID starting with dad8686854e62ab75eb62cfd0bda418ab8391cec4c1dc8ad52d4078d37f6c0bd not found: ID does not exist" Sep 15 12:00:04.270513 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.270487 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9"] Sep 15 12:00:04.283215 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:04.283180 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-97d02d-9964bb488-l5mt9"] Sep 15 12:00:05.189039 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:05.189005 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1e4751-4c95-4e92-bb96-1934ead2ffb4" path="/var/lib/kubelet/pods/6e1e4751-4c95-4e92-bb96-1934ead2ffb4/volumes" Sep 15 12:00:57.635436 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:57.635347 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw"] Sep 15 12:00:57.635980 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:57.635610 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" podUID="34861cd6-e8bb-4c4c-a2bd-fa9c29472e81" containerName="payload-processing" containerID="cri-o://a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624" gracePeriod=30 Sep 15 12:00:57.905131 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:57.905089 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 12:00:58.016179 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.016144 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jhnm\" (UniqueName: \"kubernetes.io/projected/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-kube-api-access-7jhnm\") pod \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\" (UID: \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\") " Sep 15 12:00:58.016359 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.016289 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-plugins-config-volume\") pod \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\" (UID: \"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81\") " Sep 15 12:00:58.016713 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.016685 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34861cd6-e8bb-4c4c-a2bd-fa9c29472e81" (UID: "34861cd6-e8bb-4c4c-a2bd-fa9c29472e81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:00:58.018320 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.018294 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-kube-api-access-7jhnm" (OuterVolumeSpecName: "kube-api-access-7jhnm") pod "34861cd6-e8bb-4c4c-a2bd-fa9c29472e81" (UID: "34861cd6-e8bb-4c4c-a2bd-fa9c29472e81"). InnerVolumeSpecName "kube-api-access-7jhnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:00:58.117708 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.117666 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:00:58.117708 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.117696 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jhnm\" (UniqueName: \"kubernetes.io/projected/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81-kube-api-access-7jhnm\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:00:58.556092 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.556050 2579 generic.go:356] "Generic (PLEG): container finished" podID="34861cd6-e8bb-4c4c-a2bd-fa9c29472e81" containerID="a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624" exitCode=0 Sep 15 12:00:58.556304 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.556100 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" Sep 15 12:00:58.556304 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.556116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" event={"ID":"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81","Type":"ContainerDied","Data":"a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624"} Sep 15 12:00:58.556304 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.556159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw" event={"ID":"34861cd6-e8bb-4c4c-a2bd-fa9c29472e81","Type":"ContainerDied","Data":"f30601d580acb1f33dae10805d5bbf5262fa3e07193f7d2ef6e9a21a102033a0"} Sep 15 12:00:58.556304 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.556182 2579 scope.go:117] "RemoveContainer" containerID="a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624" Sep 15 12:00:58.565582 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.565559 2579 scope.go:117] "RemoveContainer" containerID="a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624" Sep 15 12:00:58.565889 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:00:58.565868 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624\": container with ID starting with a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624 not found: ID does not exist" containerID="a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624" Sep 15 12:00:58.565969 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.565897 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624"} err="failed to get container status \"a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624\": rpc error: code = NotFound desc = could not find container \"a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624\": container with ID starting with a410dcfa70698bc2500a9c67dda33584243eb9e43c7ed22d34affb03c8560624 not found: ID does not exist" Sep 15 12:00:58.578101 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.578074 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw"] Sep 15 12:00:58.584538 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:58.584513 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e9bc87-5985fc4bc-qv7pw"] Sep 15 12:00:59.188735 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:00:59.188698 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34861cd6-e8bb-4c4c-a2bd-fa9c29472e81" path="/var/lib/kubelet/pods/34861cd6-e8bb-4c4c-a2bd-fa9c29472e81/volumes" Sep 15 12:01:13.713647 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.713605 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh"] Sep 15 12:01:13.714082 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714068 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34861cd6-e8bb-4c4c-a2bd-fa9c29472e81" containerName="payload-processing" Sep 15 12:01:13.714131 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714084 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34861cd6-e8bb-4c4c-a2bd-fa9c29472e81" containerName="payload-processing" Sep 15 12:01:13.714131 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714106 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1e4751-4c95-4e92-bb96-1934ead2ffb4" containerName="maas-api" Sep 15 12:01:13.714131 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714111 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1e4751-4c95-4e92-bb96-1934ead2ffb4" containerName="maas-api" Sep 15 12:01:13.714131 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714121 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfecf7fd-7145-4375-b904-1d876c2b8014" containerName="test-curl-7878-256646" Sep 15 12:01:13.714131 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714130 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfecf7fd-7145-4375-b904-1d876c2b8014" containerName="test-curl-7878-256646" Sep 15 12:01:13.714320 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714194 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1e4751-4c95-4e92-bb96-1934ead2ffb4" containerName="maas-api" Sep 15 12:01:13.714320 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714222 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34861cd6-e8bb-4c4c-a2bd-fa9c29472e81" containerName="payload-processing" Sep 15 12:01:13.714320 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.714231 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfecf7fd-7145-4375-b904-1d876c2b8014" containerName="test-curl-7878-256646" Sep 15 12:01:13.717564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.717543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:13.719800 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.719779 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2b3936\"" Sep 15 12:01:13.727516 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.727490 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh"] Sep 15 12:01:13.752859 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.752822 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7"] Sep 15 12:01:13.756332 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.756309 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:13.758624 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.758599 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2b3936-dockercfg-mjvwj\"" Sep 15 12:01:13.766440 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.766417 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7"] Sep 15 12:01:13.769603 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.769570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwf74\" (UniqueName: \"kubernetes.io/projected/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-kube-api-access-dwf74\") pod \"payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh\" (UID: \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:13.769764 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.769670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457ae41-dab4-46f1-beb8-ab0896c241dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7\" (UID: \"b457ae41-dab4-46f1-beb8-ab0896c241dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:13.769764 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.769753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxvs\" (UniqueName: \"kubernetes.io/projected/b457ae41-dab4-46f1-beb8-ab0896c241dc-kube-api-access-kjxvs\") pod \"payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7\" (UID: \"b457ae41-dab4-46f1-beb8-ab0896c241dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:13.769894 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.769794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh\" (UID: \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:13.870505 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.870465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457ae41-dab4-46f1-beb8-ab0896c241dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7\" (UID: \"b457ae41-dab4-46f1-beb8-ab0896c241dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:13.870739 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.870550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxvs\" (UniqueName: \"kubernetes.io/projected/b457ae41-dab4-46f1-beb8-ab0896c241dc-kube-api-access-kjxvs\") pod \"payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7\" (UID: \"b457ae41-dab4-46f1-beb8-ab0896c241dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:13.870739 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.870591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh\" (UID: \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:13.870739 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.870646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwf74\" (UniqueName: \"kubernetes.io/projected/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-kube-api-access-dwf74\") pod \"payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh\" (UID: \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:13.871336 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.871313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457ae41-dab4-46f1-beb8-ab0896c241dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7\" (UID: \"b457ae41-dab4-46f1-beb8-ab0896c241dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:13.871445 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.871418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh\" (UID: \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:13.881620 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.881593 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwf74\" (UniqueName: \"kubernetes.io/projected/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-kube-api-access-dwf74\") pod \"payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh\" (UID: \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:13.881762 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:13.881637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxvs\" (UniqueName: \"kubernetes.io/projected/b457ae41-dab4-46f1-beb8-ab0896c241dc-kube-api-access-kjxvs\") pod \"payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7\" (UID: \"b457ae41-dab4-46f1-beb8-ab0896c241dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:14.029680 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.029590 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:14.068731 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.068694 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:14.311912 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.311885 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh"] Sep 15 12:01:14.313781 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:01:14.313753 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdadc4a5_cae2_4c64_bdda_b28b45c9a54b.slice/crio-54137f3a512022e7bdf5b6c8acced8ba50860b605ed279259cce4db7bd1fc4c8 WatchSource:0}: Error finding container 54137f3a512022e7bdf5b6c8acced8ba50860b605ed279259cce4db7bd1fc4c8: Status 404 returned error can't find the container with id 54137f3a512022e7bdf5b6c8acced8ba50860b605ed279259cce4db7bd1fc4c8 Sep 15 12:01:14.364020 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.363996 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7"] Sep 15 12:01:14.372503 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:01:14.372465 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb457ae41_dab4_46f1_beb8_ab0896c241dc.slice/crio-c21692fad304f3bdd040aa35cfe3ac14f3c48d1ea4cc431965ef0d5316cfcc89 WatchSource:0}: Error finding container c21692fad304f3bdd040aa35cfe3ac14f3c48d1ea4cc431965ef0d5316cfcc89: Status 404 returned error can't find the container with id c21692fad304f3bdd040aa35cfe3ac14f3c48d1ea4cc431965ef0d5316cfcc89 Sep 15 12:01:14.647606 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.647475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" event={"ID":"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b","Type":"ContainerStarted","Data":"5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b"} Sep 15 12:01:14.647606 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.647548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" event={"ID":"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b","Type":"ContainerStarted","Data":"54137f3a512022e7bdf5b6c8acced8ba50860b605ed279259cce4db7bd1fc4c8"} Sep 15 12:01:14.647606 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.647576 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:14.649335 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.649306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" event={"ID":"b457ae41-dab4-46f1-beb8-ab0896c241dc","Type":"ContainerStarted","Data":"16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf"} Sep 15 12:01:14.649484 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.649349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" event={"ID":"b457ae41-dab4-46f1-beb8-ab0896c241dc","Type":"ContainerStarted","Data":"c21692fad304f3bdd040aa35cfe3ac14f3c48d1ea4cc431965ef0d5316cfcc89"} Sep 15 12:01:14.649484 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.649383 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:14.668627 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.668559 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" podStartSLOduration=1.668537897 podStartE2EDuration="1.668537897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:01:14.666175293 +0000 UTC m=+1362.011283901" watchObservedRunningTime="2026-09-15 12:01:14.668537897 +0000 UTC m=+1362.013646504" Sep 15 12:01:14.683682 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:14.683624 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" podStartSLOduration=1.6836081 podStartE2EDuration="1.6836081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:01:14.68215521 +0000 UTC m=+1362.027263828" watchObservedRunningTime="2026-09-15 12:01:14.6836081 +0000 UTC m=+1362.028716706" Sep 15 12:01:25.660377 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:25.660346 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:01:25.660783 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:25.660416 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:01:50.005442 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.005407 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n"] Sep 15 12:01:50.010448 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.010427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.013956 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.013932 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-32bbbb\"" Sep 15 12:01:50.014088 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.013959 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-32bbbb-dockercfg-gvfcl\"" Sep 15 12:01:50.025531 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.025504 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n"] Sep 15 12:01:50.088709 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.088682 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x"] Sep 15 12:01:50.092471 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.092455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.104661 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.104638 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x"] Sep 15 12:01:50.111150 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.111111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5edb4e4-507f-4661-8969-13090d6a0fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n\" (UID: \"b5edb4e4-507f-4661-8969-13090d6a0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.111325 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.111157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscn9\" (UniqueName: \"kubernetes.io/projected/b5edb4e4-507f-4661-8969-13090d6a0fbf-kube-api-access-bscn9\") pod \"payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n\" (UID: \"b5edb4e4-507f-4661-8969-13090d6a0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.212414 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.212373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c14dacd-ad80-485f-8dfa-a517dd6983e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x\" (UID: \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.212414 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.212417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbc77\" (UniqueName: \"kubernetes.io/projected/6c14dacd-ad80-485f-8dfa-a517dd6983e4-kube-api-access-gbc77\") pod \"payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x\" (UID: \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.212619 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.212509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5edb4e4-507f-4661-8969-13090d6a0fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n\" (UID: \"b5edb4e4-507f-4661-8969-13090d6a0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.212619 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.212538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscn9\" (UniqueName: \"kubernetes.io/projected/b5edb4e4-507f-4661-8969-13090d6a0fbf-kube-api-access-bscn9\") pod \"payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n\" (UID: \"b5edb4e4-507f-4661-8969-13090d6a0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.213186 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.213168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5edb4e4-507f-4661-8969-13090d6a0fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n\" (UID: \"b5edb4e4-507f-4661-8969-13090d6a0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.227292 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.227261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscn9\" (UniqueName: \"kubernetes.io/projected/b5edb4e4-507f-4661-8969-13090d6a0fbf-kube-api-access-bscn9\") pod \"payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n\" (UID: \"b5edb4e4-507f-4661-8969-13090d6a0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.313658 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.313580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbc77\" (UniqueName: \"kubernetes.io/projected/6c14dacd-ad80-485f-8dfa-a517dd6983e4-kube-api-access-gbc77\") pod \"payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x\" (UID: \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.313803 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.313675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c14dacd-ad80-485f-8dfa-a517dd6983e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x\" (UID: \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.314321 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.314302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c14dacd-ad80-485f-8dfa-a517dd6983e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x\" (UID: \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.320485 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.320459 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.326734 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.326708 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbc77\" (UniqueName: \"kubernetes.io/projected/6c14dacd-ad80-485f-8dfa-a517dd6983e4-kube-api-access-gbc77\") pod \"payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x\" (UID: \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.404603 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.404333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.605091 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.605067 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n"] Sep 15 12:01:50.607303 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:01:50.607275 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5edb4e4_507f_4661_8969_13090d6a0fbf.slice/crio-6de915434c8a6f17ea8cf1abdd0051f5df159abcf822c698f49b985431a11e0d WatchSource:0}: Error finding container 6de915434c8a6f17ea8cf1abdd0051f5df159abcf822c698f49b985431a11e0d: Status 404 returned error can't find the container with id 6de915434c8a6f17ea8cf1abdd0051f5df159abcf822c698f49b985431a11e0d Sep 15 12:01:50.704591 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:01:50.704562 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c14dacd_ad80_485f_8dfa_a517dd6983e4.slice/crio-ce52b63f62b95d1325e1fe2c88f97e919551f8fc1551c85f000a7c4839e5102c WatchSource:0}: Error finding container ce52b63f62b95d1325e1fe2c88f97e919551f8fc1551c85f000a7c4839e5102c: Status 404 returned error can't find the container with id ce52b63f62b95d1325e1fe2c88f97e919551f8fc1551c85f000a7c4839e5102c Sep 15 12:01:50.705474 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.705356 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x"] Sep 15 12:01:50.851746 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.851708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" event={"ID":"6c14dacd-ad80-485f-8dfa-a517dd6983e4","Type":"ContainerStarted","Data":"4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff"} Sep 15 12:01:50.851956 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.851787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" event={"ID":"6c14dacd-ad80-485f-8dfa-a517dd6983e4","Type":"ContainerStarted","Data":"ce52b63f62b95d1325e1fe2c88f97e919551f8fc1551c85f000a7c4839e5102c"} Sep 15 12:01:50.851956 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.851863 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:01:50.853587 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.853560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" event={"ID":"b5edb4e4-507f-4661-8969-13090d6a0fbf","Type":"ContainerStarted","Data":"83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad"} Sep 15 12:01:50.853708 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.853615 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:01:50.853708 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.853638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" event={"ID":"b5edb4e4-507f-4661-8969-13090d6a0fbf","Type":"ContainerStarted","Data":"6de915434c8a6f17ea8cf1abdd0051f5df159abcf822c698f49b985431a11e0d"} Sep 15 12:01:50.931257 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.931195 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" podStartSLOduration=1.9311800799999999 podStartE2EDuration="1.93118008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:01:50.930769136 +0000 UTC m=+1398.275877746" watchObservedRunningTime="2026-09-15 12:01:50.93118008 +0000 UTC m=+1398.276288685" Sep 15 12:01:50.931429 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:01:50.931372 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" podStartSLOduration=0.931365316 podStartE2EDuration="931.365316ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:01:50.902607653 +0000 UTC m=+1398.247716262" watchObservedRunningTime="2026-09-15 12:01:50.931365316 +0000 UTC m=+1398.276473922" Sep 15 12:02:01.861301 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:01.861259 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:02:01.861697 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:01.861556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:02:19.649077 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:19.649041 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb"] Sep 15 12:02:34.670844 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:34.670745 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" podUID="a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" containerName="main" containerID="cri-o://1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc" gracePeriod=45 Sep 15 12:02:34.924851 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:34.924791 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 12:02:35.045269 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045227 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-home\") pod \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " Sep 15 12:02:35.045428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045291 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k86hf\" (UniqueName: \"kubernetes.io/projected/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-kube-api-access-k86hf\") pod \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " Sep 15 12:02:35.045428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045345 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-dshm\") pod \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " Sep 15 12:02:35.045428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045379 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tmp-dir\") pod \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " Sep 15 12:02:35.045428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045396 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tls-certs\") pod \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " Sep 15 12:02:35.045669 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045468 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-model-cache\") pod \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\" (UID: \"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8\") " Sep 15 12:02:35.045669 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045506 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-home" (OuterVolumeSpecName: "home") pod "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" (UID: "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:02:35.045669 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045656 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" (UID: "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:02:35.045815 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045716 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-home\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:02:35.045815 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.045736 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" (UID: "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:02:35.047837 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.047815 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-dshm" (OuterVolumeSpecName: "dshm") pod "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" (UID: "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:02:35.047837 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.047813 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" (UID: "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:02:35.047837 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.047828 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-kube-api-access-k86hf" (OuterVolumeSpecName: "kube-api-access-k86hf") pod "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" (UID: "a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8"). InnerVolumeSpecName "kube-api-access-k86hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:02:35.098461 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.098419 2579 generic.go:356] "Generic (PLEG): container finished" podID="a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" containerID="1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc" exitCode=0 Sep 15 12:02:35.098657 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.098475 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" Sep 15 12:02:35.098657 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.098506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" event={"ID":"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8","Type":"ContainerDied","Data":"1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc"} Sep 15 12:02:35.098657 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.098559 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb" event={"ID":"a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8","Type":"ContainerDied","Data":"41fac9554f2efe45e5aac988ad6d8ad5cf3a6d4a2e78d39c560dfdc175fd3668"} Sep 15 12:02:35.098657 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.098581 2579 scope.go:117] "RemoveContainer" containerID="1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc" Sep 15 12:02:35.109897 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.109878 2579 scope.go:117] "RemoveContainer" containerID="1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc" Sep 15 12:02:35.110261 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:02:35.110240 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc\": container with ID starting with 1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc not found: ID does not exist" containerID="1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc" Sep 15 12:02:35.110335 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.110274 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc"} err="failed to get container status \"1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc\": rpc error: code = NotFound desc = could not find container \"1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc\": container with ID starting with 1faa45243fb346892048a954b5e7b032ce0207d0e0fad4678811ff77aea603dc not found: ID does not exist" Sep 15 12:02:35.125313 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.125284 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb"] Sep 15 12:02:35.129671 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.129645 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ac76e5/e2e-conflict-a-6fb44257-kserve-9b7c5df97-ssnzb"] Sep 15 12:02:35.146567 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.146545 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-dshm\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:02:35.146672 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.146590 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tmp-dir\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:02:35.146672 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.146604 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-tls-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:02:35.146672 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.146615 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-model-cache\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:02:35.146672 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.146629 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k86hf\" (UniqueName: \"kubernetes.io/projected/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8-kube-api-access-k86hf\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:02:35.189261 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:02:35.189223 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" path="/var/lib/kubelet/pods/a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8/volumes" Sep 15 12:03:33.217310 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:03:33.217275 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:03:33.219164 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:03:33.219140 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:03:33.353869 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:03:33.353837 2579 scope.go:117] "RemoveContainer" containerID="9c696dd95b60dd8fdb0af4a8e98420313e9866d54ef46e3e4ef77867835a6b5e" Sep 15 12:04:21.357334 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.357293 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn"] Sep 15 12:04:21.357873 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.357856 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" containerName="main" Sep 15 12:04:21.357873 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.357875 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" containerName="main" Sep 15 12:04:21.358007 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.357995 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1c7cd2d-aef8-4bb3-b3a4-e2dcaded56e8" containerName="main" Sep 15 12:04:21.361151 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.361131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.363355 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.363331 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-25d1cb-serving-cert\"" Sep 15 12:04:21.372264 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.372235 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn"] Sep 15 12:04:21.410348 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.410300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebf51ba-2929-4c79-a662-f0822c10f76b-openshift-service-ca\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.410556 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.410363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.410556 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.410471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-metrics-certs\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.410556 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.410499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h45l\" (UniqueName: \"kubernetes.io/projected/5ebf51ba-2929-4c79-a662-f0822c10f76b-kube-api-access-5h45l\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.511024 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.510985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebf51ba-2929-4c79-a662-f0822c10f76b-openshift-service-ca\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.511024 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.511032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.511352 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.511074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-metrics-certs\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.511352 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:04:21.511242 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-serving-cert: secret "maas-api-e2e-api-a-25d1cb-serving-cert" not found Sep 15 12:04:21.511352 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.511256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h45l\" (UniqueName: \"kubernetes.io/projected/5ebf51ba-2929-4c79-a662-f0822c10f76b-kube-api-access-5h45l\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.511352 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:04:21.511331 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls podName:5ebf51ba-2929-4c79-a662-f0822c10f76b nodeName:}" failed. No retries permitted until 2026-09-15 12:04:22.011306571 +0000 UTC m=+1549.356415170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls") pod "maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" (UID: "5ebf51ba-2929-4c79-a662-f0822c10f76b") : secret "maas-api-e2e-api-a-25d1cb-serving-cert" not found Sep 15 12:04:21.511727 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.511705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebf51ba-2929-4c79-a662-f0822c10f76b-openshift-service-ca\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.513867 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.513845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-metrics-certs\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:21.522319 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:21.522296 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h45l\" (UniqueName: \"kubernetes.io/projected/5ebf51ba-2929-4c79-a662-f0822c10f76b-kube-api-access-5h45l\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:22.016586 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:22.016550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:22.019166 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:22.019130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls\") pod \"maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:22.274543 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:22.274450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:22.560704 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:22.560525 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn"] Sep 15 12:04:22.563355 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:04:22.563323 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebf51ba_2929_4c79_a662_f0822c10f76b.slice/crio-052e294ddbc6aa14aecceccb771a5ec88a41cb5ee3ac0479e84a847177d5c1e8 WatchSource:0}: Error finding container 052e294ddbc6aa14aecceccb771a5ec88a41cb5ee3ac0479e84a847177d5c1e8: Status 404 returned error can't find the container with id 052e294ddbc6aa14aecceccb771a5ec88a41cb5ee3ac0479e84a847177d5c1e8 Sep 15 12:04:22.564817 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:22.564799 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:04:22.698579 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:22.698497 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" event={"ID":"5ebf51ba-2929-4c79-a662-f0822c10f76b","Type":"ContainerStarted","Data":"052e294ddbc6aa14aecceccb771a5ec88a41cb5ee3ac0479e84a847177d5c1e8"} Sep 15 12:04:23.707799 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:23.707759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" event={"ID":"5ebf51ba-2929-4c79-a662-f0822c10f76b","Type":"ContainerStarted","Data":"0ae983b93b5543c1cb9e4026672df71d6c0b0e75bbd9dd2b579acc9e89a66af4"} Sep 15 12:04:23.708265 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:23.707961 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:23.725527 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:23.725478 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" podStartSLOduration=2.192073083 podStartE2EDuration="2.725461225s" totalImagesPullingTime="533.388142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:04:23.724810544 +0000 UTC m=+1551.069919151" watchObservedRunningTime="2026-09-15 12:04:23.725461225 +0000 UTC m=+1551.070569831" Sep 15 12:04:29.718666 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:29.718638 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:04:52.811013 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:52.810980 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q"] Sep 15 12:04:52.814781 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:52.814760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:52.818700 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:52.818672 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0848f9\"" Sep 15 12:04:52.818992 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:52.818971 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0848f9-dockercfg-wfd6c\"" Sep 15 12:04:52.827448 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:52.827422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q"] Sep 15 12:04:52.915038 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:52.914999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xbbs\" (UniqueName: \"kubernetes.io/projected/d56dfb2b-e546-487c-a86e-0cff0c657022-kube-api-access-2xbbs\") pod \"payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q\" (UID: \"d56dfb2b-e546-487c-a86e-0cff0c657022\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:52.915278 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:52.915091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56dfb2b-e546-487c-a86e-0cff0c657022-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q\" (UID: \"d56dfb2b-e546-487c-a86e-0cff0c657022\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:53.016677 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.016634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xbbs\" (UniqueName: \"kubernetes.io/projected/d56dfb2b-e546-487c-a86e-0cff0c657022-kube-api-access-2xbbs\") pod \"payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q\" (UID: \"d56dfb2b-e546-487c-a86e-0cff0c657022\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:53.016851 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.016714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56dfb2b-e546-487c-a86e-0cff0c657022-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q\" (UID: \"d56dfb2b-e546-487c-a86e-0cff0c657022\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:53.017497 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.017478 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56dfb2b-e546-487c-a86e-0cff0c657022-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q\" (UID: \"d56dfb2b-e546-487c-a86e-0cff0c657022\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:53.029089 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.029059 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xbbs\" (UniqueName: \"kubernetes.io/projected/d56dfb2b-e546-487c-a86e-0cff0c657022-kube-api-access-2xbbs\") pod \"payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q\" (UID: \"d56dfb2b-e546-487c-a86e-0cff0c657022\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:53.125555 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.125454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:53.409281 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.409257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q"] Sep 15 12:04:53.411897 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:04:53.411871 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56dfb2b_e546_487c_a86e_0cff0c657022.slice/crio-6b1e2f45e6771a1f4d89a657e5ffdfcfeabe4876eb62217b5aa7ee0c619fa514 WatchSource:0}: Error finding container 6b1e2f45e6771a1f4d89a657e5ffdfcfeabe4876eb62217b5aa7ee0c619fa514: Status 404 returned error can't find the container with id 6b1e2f45e6771a1f4d89a657e5ffdfcfeabe4876eb62217b5aa7ee0c619fa514 Sep 15 12:04:53.872161 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.872116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" event={"ID":"d56dfb2b-e546-487c-a86e-0cff0c657022","Type":"ContainerStarted","Data":"8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb"} Sep 15 12:04:53.872621 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.872181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" event={"ID":"d56dfb2b-e546-487c-a86e-0cff0c657022","Type":"ContainerStarted","Data":"6b1e2f45e6771a1f4d89a657e5ffdfcfeabe4876eb62217b5aa7ee0c619fa514"} Sep 15 12:04:53.872621 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.872315 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:04:53.894458 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:04:53.894400 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" podStartSLOduration=1.894380202 podStartE2EDuration="1.894380202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:04:53.892938662 +0000 UTC m=+1581.238047265" watchObservedRunningTime="2026-09-15 12:04:53.894380202 +0000 UTC m=+1581.239488842" Sep 15 12:05:04.880192 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:04.880157 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:05:23.770449 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:23.770412 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn"] Sep 15 12:05:23.770955 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:23.770710 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" podUID="5ebf51ba-2929-4c79-a662-f0822c10f76b" containerName="maas-api" containerID="cri-o://0ae983b93b5543c1cb9e4026672df71d6c0b0e75bbd9dd2b579acc9e89a66af4" gracePeriod=30 Sep 15 12:05:24.052187 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.052153 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ebf51ba-2929-4c79-a662-f0822c10f76b" containerID="0ae983b93b5543c1cb9e4026672df71d6c0b0e75bbd9dd2b579acc9e89a66af4" exitCode=0 Sep 15 12:05:24.052428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.052212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" event={"ID":"5ebf51ba-2929-4c79-a662-f0822c10f76b","Type":"ContainerDied","Data":"0ae983b93b5543c1cb9e4026672df71d6c0b0e75bbd9dd2b579acc9e89a66af4"} Sep 15 12:05:24.052428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.052302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" event={"ID":"5ebf51ba-2929-4c79-a662-f0822c10f76b","Type":"ContainerDied","Data":"052e294ddbc6aa14aecceccb771a5ec88a41cb5ee3ac0479e84a847177d5c1e8"} Sep 15 12:05:24.052428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.052322 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="052e294ddbc6aa14aecceccb771a5ec88a41cb5ee3ac0479e84a847177d5c1e8" Sep 15 12:05:24.058867 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.058843 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:05:24.209946 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.209857 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebf51ba-2929-4c79-a662-f0822c10f76b-openshift-service-ca\") pod \"5ebf51ba-2929-4c79-a662-f0822c10f76b\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " Sep 15 12:05:24.209946 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.209897 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-metrics-certs\") pod \"5ebf51ba-2929-4c79-a662-f0822c10f76b\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " Sep 15 12:05:24.210303 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.209972 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls\") pod \"5ebf51ba-2929-4c79-a662-f0822c10f76b\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " Sep 15 12:05:24.210303 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.210171 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h45l\" (UniqueName: \"kubernetes.io/projected/5ebf51ba-2929-4c79-a662-f0822c10f76b-kube-api-access-5h45l\") pod \"5ebf51ba-2929-4c79-a662-f0822c10f76b\" (UID: \"5ebf51ba-2929-4c79-a662-f0822c10f76b\") " Sep 15 12:05:24.210425 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.210349 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebf51ba-2929-4c79-a662-f0822c10f76b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ebf51ba-2929-4c79-a662-f0822c10f76b" (UID: "5ebf51ba-2929-4c79-a662-f0822c10f76b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:24.210592 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.210564 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebf51ba-2929-4c79-a662-f0822c10f76b-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:05:24.212353 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.212320 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ebf51ba-2929-4c79-a662-f0822c10f76b" (UID: "5ebf51ba-2929-4c79-a662-f0822c10f76b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:05:24.212461 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.212451 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ebf51ba-2929-4c79-a662-f0822c10f76b" (UID: "5ebf51ba-2929-4c79-a662-f0822c10f76b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:05:24.212506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.212458 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebf51ba-2929-4c79-a662-f0822c10f76b-kube-api-access-5h45l" (OuterVolumeSpecName: "kube-api-access-5h45l") pod "5ebf51ba-2929-4c79-a662-f0822c10f76b" (UID: "5ebf51ba-2929-4c79-a662-f0822c10f76b"). InnerVolumeSpecName "kube-api-access-5h45l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:05:24.312071 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.311961 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h45l\" (UniqueName: \"kubernetes.io/projected/5ebf51ba-2929-4c79-a662-f0822c10f76b-kube-api-access-5h45l\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:05:24.312071 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.312004 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:05:24.312071 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:24.312018 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebf51ba-2929-4c79-a662-f0822c10f76b-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:05:25.057710 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:25.057673 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn" Sep 15 12:05:25.082961 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:25.082927 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn"] Sep 15 12:05:25.085312 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:25.085285 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-25d1cb-55565b6d6d-h6nfn"] Sep 15 12:05:25.189383 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:25.189349 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ebf51ba-2929-4c79-a662-f0822c10f76b" path="/var/lib/kubelet/pods/5ebf51ba-2929-4c79-a662-f0822c10f76b/volumes" Sep 15 12:05:36.128399 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:36.128298 2579 generic.go:356] "Generic (PLEG): container finished" podID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerID="c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0" exitCode=137 Sep 15 12:05:36.128902 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:36.128397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" event={"ID":"1b8589f0-8c9f-47eb-ba14-4e9578f13774","Type":"ContainerDied","Data":"c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0"} Sep 15 12:05:36.128902 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:36.128806 2579 scope.go:117] "RemoveContainer" containerID="c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0" Sep 15 12:05:37.138541 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:37.138499 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" event={"ID":"1b8589f0-8c9f-47eb-ba14-4e9578f13774","Type":"ContainerStarted","Data":"6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2"} Sep 15 12:05:37.139008 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:37.138710 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 12:05:47.462624 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.462581 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q"] Sep 15 12:05:47.463516 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.463459 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" podUID="d56dfb2b-e546-487c-a86e-0cff0c657022" containerName="payload-pre-processing" containerID="cri-o://8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb" gracePeriod=30 Sep 15 12:05:47.719157 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.719092 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:05:47.728878 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.728851 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56dfb2b-e546-487c-a86e-0cff0c657022-plugins-config-volume\") pod \"d56dfb2b-e546-487c-a86e-0cff0c657022\" (UID: \"d56dfb2b-e546-487c-a86e-0cff0c657022\") " Sep 15 12:05:47.729016 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.728887 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xbbs\" (UniqueName: \"kubernetes.io/projected/d56dfb2b-e546-487c-a86e-0cff0c657022-kube-api-access-2xbbs\") pod \"d56dfb2b-e546-487c-a86e-0cff0c657022\" (UID: \"d56dfb2b-e546-487c-a86e-0cff0c657022\") " Sep 15 12:05:47.729310 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.729285 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56dfb2b-e546-487c-a86e-0cff0c657022-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d56dfb2b-e546-487c-a86e-0cff0c657022" (UID: "d56dfb2b-e546-487c-a86e-0cff0c657022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:05:47.731343 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.731317 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56dfb2b-e546-487c-a86e-0cff0c657022-kube-api-access-2xbbs" (OuterVolumeSpecName: "kube-api-access-2xbbs") pod "d56dfb2b-e546-487c-a86e-0cff0c657022" (UID: "d56dfb2b-e546-487c-a86e-0cff0c657022"). InnerVolumeSpecName "kube-api-access-2xbbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:05:47.829725 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.829687 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d56dfb2b-e546-487c-a86e-0cff0c657022-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:05:47.829725 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:47.829722 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xbbs\" (UniqueName: \"kubernetes.io/projected/d56dfb2b-e546-487c-a86e-0cff0c657022-kube-api-access-2xbbs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:05:48.146885 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.146798 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 12:05:48.201628 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.201591 2579 generic.go:356] "Generic (PLEG): container finished" podID="d56dfb2b-e546-487c-a86e-0cff0c657022" containerID="8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb" exitCode=0 Sep 15 12:05:48.201897 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.201876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" event={"ID":"d56dfb2b-e546-487c-a86e-0cff0c657022","Type":"ContainerDied","Data":"8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb"} Sep 15 12:05:48.202096 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.202081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" event={"ID":"d56dfb2b-e546-487c-a86e-0cff0c657022","Type":"ContainerDied","Data":"6b1e2f45e6771a1f4d89a657e5ffdfcfeabe4876eb62217b5aa7ee0c619fa514"} Sep 15 12:05:48.202298 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.202248 2579 scope.go:117] "RemoveContainer" containerID="8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb" Sep 15 12:05:48.202456 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.202425 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q" Sep 15 12:05:48.216235 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.216215 2579 scope.go:117] "RemoveContainer" containerID="8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb" Sep 15 12:05:48.216660 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:05:48.216632 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb\": container with ID starting with 8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb not found: ID does not exist" containerID="8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb" Sep 15 12:05:48.216731 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.216674 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb"} err="failed to get container status \"8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb\": rpc error: code = NotFound desc = could not find container \"8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb\": container with ID starting with 8f81464ec811782a924e91e1c30bf1c81d05632e418cb2151265aeac05d525bb not found: ID does not exist" Sep 15 12:05:48.232835 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.232782 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q"] Sep 15 12:05:48.239893 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:48.239856 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0848f9-9bd7d447d-pv86q"] Sep 15 12:05:49.189928 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:05:49.189891 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d56dfb2b-e546-487c-a86e-0cff0c657022" path="/var/lib/kubelet/pods/d56dfb2b-e546-487c-a86e-0cff0c657022/volumes" Sep 15 12:06:06.248701 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.248658 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w"] Sep 15 12:06:06.249264 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.248953 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" podUID="f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" containerName="maas-api" containerID="cri-o://badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8" gracePeriod=30 Sep 15 12:06:06.311726 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.311660 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq"] Sep 15 12:06:06.311973 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.311946 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" podUID="da0b566f-95d3-4f6d-8a8c-339e356a3013" containerName="payload-processing" containerID="cri-o://40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735" gracePeriod=30 Sep 15 12:06:06.627761 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.627725 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 12:06:06.634021 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.633980 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 12:06:06.715241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.715180 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-maas-api-tls\") pod \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " Sep 15 12:06:06.715447 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.715309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-metrics-certs\") pod \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " Sep 15 12:06:06.715447 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.715350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-openshift-service-ca\") pod \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " Sep 15 12:06:06.715447 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.715392 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b566f-95d3-4f6d-8a8c-339e356a3013-plugins-config-volume\") pod \"da0b566f-95d3-4f6d-8a8c-339e356a3013\" (UID: \"da0b566f-95d3-4f6d-8a8c-339e356a3013\") " Sep 15 12:06:06.715643 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.715478 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw886\" (UniqueName: \"kubernetes.io/projected/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-kube-api-access-lw886\") pod \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\" (UID: \"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8\") " Sep 15 12:06:06.715643 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.715501 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpdtg\" (UniqueName: \"kubernetes.io/projected/da0b566f-95d3-4f6d-8a8c-339e356a3013-kube-api-access-hpdtg\") pod \"da0b566f-95d3-4f6d-8a8c-339e356a3013\" (UID: \"da0b566f-95d3-4f6d-8a8c-339e356a3013\") " Sep 15 12:06:06.716031 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.716003 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" (UID: "f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:06:06.716499 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.716472 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0b566f-95d3-4f6d-8a8c-339e356a3013-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da0b566f-95d3-4f6d-8a8c-339e356a3013" (UID: "da0b566f-95d3-4f6d-8a8c-339e356a3013"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:06:06.718612 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.718579 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" (UID: "f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:06:06.718723 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.718665 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-kube-api-access-lw886" (OuterVolumeSpecName: "kube-api-access-lw886") pod "f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" (UID: "f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8"). InnerVolumeSpecName "kube-api-access-lw886". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:06:06.719099 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.719071 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0b566f-95d3-4f6d-8a8c-339e356a3013-kube-api-access-hpdtg" (OuterVolumeSpecName: "kube-api-access-hpdtg") pod "da0b566f-95d3-4f6d-8a8c-339e356a3013" (UID: "da0b566f-95d3-4f6d-8a8c-339e356a3013"). InnerVolumeSpecName "kube-api-access-hpdtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:06:06.720166 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.720137 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" (UID: "f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:06:06.816934 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.816830 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0b566f-95d3-4f6d-8a8c-339e356a3013-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:06.816934 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.816866 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw886\" (UniqueName: \"kubernetes.io/projected/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-kube-api-access-lw886\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:06.816934 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.816879 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpdtg\" (UniqueName: \"kubernetes.io/projected/da0b566f-95d3-4f6d-8a8c-339e356a3013-kube-api-access-hpdtg\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:06.816934 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.816893 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:06.816934 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.816904 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:06.816934 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:06.816916 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:07.317758 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.317725 2579 generic.go:356] "Generic (PLEG): container finished" podID="da0b566f-95d3-4f6d-8a8c-339e356a3013" containerID="40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735" exitCode=0 Sep 15 12:06:07.318231 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.317778 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" Sep 15 12:06:07.318231 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.317819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" event={"ID":"da0b566f-95d3-4f6d-8a8c-339e356a3013","Type":"ContainerDied","Data":"40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735"} Sep 15 12:06:07.318231 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.317875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq" event={"ID":"da0b566f-95d3-4f6d-8a8c-339e356a3013","Type":"ContainerDied","Data":"fc1a5a4d99148d9fd87a3693424aff9b11ef8605ae35f414f836f6e27cfb348a"} Sep 15 12:06:07.318231 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.317907 2579 scope.go:117] "RemoveContainer" containerID="40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735" Sep 15 12:06:07.320066 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.320030 2579 generic.go:356] "Generic (PLEG): container finished" podID="f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" containerID="badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8" exitCode=0 Sep 15 12:06:07.320240 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.320160 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" event={"ID":"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8","Type":"ContainerDied","Data":"badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8"} Sep 15 12:06:07.320319 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.320239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" event={"ID":"f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8","Type":"ContainerDied","Data":"6009da2b2092a80bf20872997ba85f3728f55ea5008e01cc4304eb26fc66f1a3"} Sep 15 12:06:07.320395 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.320372 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w" Sep 15 12:06:07.329011 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.328991 2579 scope.go:117] "RemoveContainer" containerID="40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735" Sep 15 12:06:07.329451 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:06:07.329427 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735\": container with ID starting with 40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735 not found: ID does not exist" containerID="40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735" Sep 15 12:06:07.329554 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.329462 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735"} err="failed to get container status \"40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735\": rpc error: code = NotFound desc = could not find container \"40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735\": container with ID starting with 40666cc2bff6deda2504d5db36383f37306337e082d543ec4131d07910352735 not found: ID does not exist" Sep 15 12:06:07.329554 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.329489 2579 scope.go:117] "RemoveContainer" containerID="badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8" Sep 15 12:06:07.338708 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.338688 2579 scope.go:117] "RemoveContainer" containerID="badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8" Sep 15 12:06:07.339029 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:06:07.339009 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8\": container with ID starting with badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8 not found: ID does not exist" containerID="badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8" Sep 15 12:06:07.339099 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.339038 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8"} err="failed to get container status \"badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8\": rpc error: code = NotFound desc = could not find container \"badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8\": container with ID starting with badea85622f28b0096d445f887bef835ef2bad588aa4659087952c562a571cf8 not found: ID does not exist" Sep 15 12:06:07.340167 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.340140 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq"] Sep 15 12:06:07.344789 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.344760 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-ec8230-68db66c887-x22dq"] Sep 15 12:06:07.356481 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.356452 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w"] Sep 15 12:06:07.360953 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:07.360925 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ec8230-5967c6977f-4tt2w"] Sep 15 12:06:09.189464 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:09.189431 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da0b566f-95d3-4f6d-8a8c-339e356a3013" path="/var/lib/kubelet/pods/da0b566f-95d3-4f6d-8a8c-339e356a3013/volumes" Sep 15 12:06:09.189846 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:09.189825 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" path="/var/lib/kubelet/pods/f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8/volumes" Sep 15 12:06:24.332045 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.331987 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6"] Sep 15 12:06:24.332665 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.332310 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" podUID="6d2797a9-934a-4be7-9fd5-76faf0cf3b35" containerName="maas-api" containerID="cri-o://5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2" gracePeriod=30 Sep 15 12:06:24.441934 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.441895 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch"] Sep 15 12:06:24.442218 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.442174 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" podUID="64fbe269-8c48-4364-9721-6a5db61f69ab" containerName="payload-processing" containerID="cri-o://8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998" gracePeriod=30 Sep 15 12:06:24.630968 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.630937 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 12:06:24.717360 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.717328 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 12:06:24.783373 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.783340 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42fnv\" (UniqueName: \"kubernetes.io/projected/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-kube-api-access-42fnv\") pod \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " Sep 15 12:06:24.783373 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.783385 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-maas-api-tls\") pod \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " Sep 15 12:06:24.783624 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.783415 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-openshift-service-ca\") pod \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " Sep 15 12:06:24.783624 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.783478 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-metrics-certs\") pod \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\" (UID: \"6d2797a9-934a-4be7-9fd5-76faf0cf3b35\") " Sep 15 12:06:24.783826 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.783801 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d2797a9-934a-4be7-9fd5-76faf0cf3b35" (UID: "6d2797a9-934a-4be7-9fd5-76faf0cf3b35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:06:24.785809 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.785768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d2797a9-934a-4be7-9fd5-76faf0cf3b35" (UID: "6d2797a9-934a-4be7-9fd5-76faf0cf3b35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:06:24.785809 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.785791 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-kube-api-access-42fnv" (OuterVolumeSpecName: "kube-api-access-42fnv") pod "6d2797a9-934a-4be7-9fd5-76faf0cf3b35" (UID: "6d2797a9-934a-4be7-9fd5-76faf0cf3b35"). InnerVolumeSpecName "kube-api-access-42fnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:06:24.785981 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.785844 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d2797a9-934a-4be7-9fd5-76faf0cf3b35" (UID: "6d2797a9-934a-4be7-9fd5-76faf0cf3b35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:06:24.884300 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.884210 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9r7r\" (UniqueName: \"kubernetes.io/projected/64fbe269-8c48-4364-9721-6a5db61f69ab-kube-api-access-g9r7r\") pod \"64fbe269-8c48-4364-9721-6a5db61f69ab\" (UID: \"64fbe269-8c48-4364-9721-6a5db61f69ab\") " Sep 15 12:06:24.884300 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.884258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbe269-8c48-4364-9721-6a5db61f69ab-plugins-config-volume\") pod \"64fbe269-8c48-4364-9721-6a5db61f69ab\" (UID: \"64fbe269-8c48-4364-9721-6a5db61f69ab\") " Sep 15 12:06:24.884487 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.884424 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:24.884487 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.884434 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42fnv\" (UniqueName: \"kubernetes.io/projected/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-kube-api-access-42fnv\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:24.884487 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.884443 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:24.884487 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.884452 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d2797a9-934a-4be7-9fd5-76faf0cf3b35-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:24.884660 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.884632 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fbe269-8c48-4364-9721-6a5db61f69ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64fbe269-8c48-4364-9721-6a5db61f69ab" (UID: "64fbe269-8c48-4364-9721-6a5db61f69ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:06:24.886637 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.886606 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fbe269-8c48-4364-9721-6a5db61f69ab-kube-api-access-g9r7r" (OuterVolumeSpecName: "kube-api-access-g9r7r") pod "64fbe269-8c48-4364-9721-6a5db61f69ab" (UID: "64fbe269-8c48-4364-9721-6a5db61f69ab"). InnerVolumeSpecName "kube-api-access-g9r7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:06:24.985173 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.985124 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9r7r\" (UniqueName: \"kubernetes.io/projected/64fbe269-8c48-4364-9721-6a5db61f69ab-kube-api-access-g9r7r\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:24.985173 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:24.985161 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64fbe269-8c48-4364-9721-6a5db61f69ab-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:06:25.428092 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.428059 2579 generic.go:356] "Generic (PLEG): container finished" podID="64fbe269-8c48-4364-9721-6a5db61f69ab" containerID="8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998" exitCode=0 Sep 15 12:06:25.428618 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.428109 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" Sep 15 12:06:25.428618 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.428151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" event={"ID":"64fbe269-8c48-4364-9721-6a5db61f69ab","Type":"ContainerDied","Data":"8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998"} Sep 15 12:06:25.428618 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.428213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch" event={"ID":"64fbe269-8c48-4364-9721-6a5db61f69ab","Type":"ContainerDied","Data":"b699387ee80a3ba0740001fe1ab9d39b3c19224691b62aadd35570eb56cfbdec"} Sep 15 12:06:25.428618 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.428227 2579 scope.go:117] "RemoveContainer" containerID="8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998" Sep 15 12:06:25.429826 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.429804 2579 generic.go:356] "Generic (PLEG): container finished" podID="6d2797a9-934a-4be7-9fd5-76faf0cf3b35" containerID="5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2" exitCode=0 Sep 15 12:06:25.429936 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.429853 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" Sep 15 12:06:25.429936 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.429869 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" event={"ID":"6d2797a9-934a-4be7-9fd5-76faf0cf3b35","Type":"ContainerDied","Data":"5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2"} Sep 15 12:06:25.429936 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.429927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6" event={"ID":"6d2797a9-934a-4be7-9fd5-76faf0cf3b35","Type":"ContainerDied","Data":"d2d1c2d9589af0e0cb70d23d52f12b40a757ed4f4560e8727892ae1be9f39b8b"} Sep 15 12:06:25.437808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.437787 2579 scope.go:117] "RemoveContainer" containerID="8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998" Sep 15 12:06:25.438162 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:06:25.438140 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998\": container with ID starting with 8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998 not found: ID does not exist" containerID="8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998" Sep 15 12:06:25.438301 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.438174 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998"} err="failed to get container status \"8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998\": rpc error: code = NotFound desc = could not find container \"8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998\": container with ID starting with 8875376cf489a167be06ca82420491d53941b82032c22d7f69f63b425f7ae998 not found: ID does not exist" Sep 15 12:06:25.438301 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.438231 2579 scope.go:117] "RemoveContainer" containerID="5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2" Sep 15 12:06:25.447241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.447190 2579 scope.go:117] "RemoveContainer" containerID="5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2" Sep 15 12:06:25.447626 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:06:25.447597 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2\": container with ID starting with 5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2 not found: ID does not exist" containerID="5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2" Sep 15 12:06:25.447753 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.447637 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2"} err="failed to get container status \"5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2\": rpc error: code = NotFound desc = could not find container \"5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2\": container with ID starting with 5bdd243889a9328e93cdd51a3eb55e09e98d7ab1b14fea3041cbce051bc0c8d2 not found: ID does not exist" Sep 15 12:06:25.450017 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.449995 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6"] Sep 15 12:06:25.460612 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.460540 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-6dca39-59db87b6d9-x87h6"] Sep 15 12:06:25.472719 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.472685 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch"] Sep 15 12:06:25.478354 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:25.478326 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6dca39-677c566b6c-89jch"] Sep 15 12:06:27.188742 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:27.188696 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fbe269-8c48-4364-9721-6a5db61f69ab" path="/var/lib/kubelet/pods/64fbe269-8c48-4364-9721-6a5db61f69ab/volumes" Sep 15 12:06:27.189320 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:06:27.189299 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d2797a9-934a-4be7-9fd5-76faf0cf3b35" path="/var/lib/kubelet/pods/6d2797a9-934a-4be7-9fd5-76faf0cf3b35/volumes" Sep 15 12:07:02.372975 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.372881 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf"] Sep 15 12:07:02.373553 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.373227 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" podUID="cd7cd0ba-349a-48b6-9577-606aac4f8068" containerName="maas-api" containerID="cri-o://cc51485b2ac446c89572fdac8b7722f7b342d51d27abcd2292cc84b29093d9e4" gracePeriod=30 Sep 15 12:07:02.436982 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.436944 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm"] Sep 15 12:07:02.437265 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.437173 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" podUID="a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e" containerName="payload-processing" containerID="cri-o://9d89637aa585c570b0a474ba181aee11d5923fe9ca844a331e012a1639245d2c" gracePeriod=30 Sep 15 12:07:02.448418 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.448382 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt"] Sep 15 12:07:02.448682 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.448631 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" podUID="aa2daa27-f5c0-4458-bb7d-184d9e40a907" containerName="payload-pre-processing" containerID="cri-o://e23eeddb6cf93d291ecb415bed10fb77adfe1c4f8ca67547a06e01dfa72a623a" gracePeriod=30 Sep 15 12:07:02.647295 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.647015 2579 generic.go:356] "Generic (PLEG): container finished" podID="aa2daa27-f5c0-4458-bb7d-184d9e40a907" containerID="e23eeddb6cf93d291ecb415bed10fb77adfe1c4f8ca67547a06e01dfa72a623a" exitCode=0 Sep 15 12:07:02.647295 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.647174 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" event={"ID":"aa2daa27-f5c0-4458-bb7d-184d9e40a907","Type":"ContainerDied","Data":"e23eeddb6cf93d291ecb415bed10fb77adfe1c4f8ca67547a06e01dfa72a623a"} Sep 15 12:07:02.648944 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.648913 2579 generic.go:356] "Generic (PLEG): container finished" podID="a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e" containerID="9d89637aa585c570b0a474ba181aee11d5923fe9ca844a331e012a1639245d2c" exitCode=0 Sep 15 12:07:02.649108 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.649015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" event={"ID":"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e","Type":"ContainerDied","Data":"9d89637aa585c570b0a474ba181aee11d5923fe9ca844a331e012a1639245d2c"} Sep 15 12:07:02.651101 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.651073 2579 generic.go:356] "Generic (PLEG): container finished" podID="cd7cd0ba-349a-48b6-9577-606aac4f8068" containerID="cc51485b2ac446c89572fdac8b7722f7b342d51d27abcd2292cc84b29093d9e4" exitCode=0 Sep 15 12:07:02.651262 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.651121 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" event={"ID":"cd7cd0ba-349a-48b6-9577-606aac4f8068","Type":"ContainerDied","Data":"cc51485b2ac446c89572fdac8b7722f7b342d51d27abcd2292cc84b29093d9e4"} Sep 15 12:07:02.777477 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.777449 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 12:07:02.780672 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.780641 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 12:07:02.784688 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.784660 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 12:07:02.806448 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.806416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-metrics-certs\") pod \"cd7cd0ba-349a-48b6-9577-606aac4f8068\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " Sep 15 12:07:02.806652 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.806478 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls\") pod \"cd7cd0ba-349a-48b6-9577-606aac4f8068\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " Sep 15 12:07:02.806652 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.806515 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-plugins-config-volume\") pod \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\" (UID: \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\") " Sep 15 12:07:02.806652 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.806541 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk9rj\" (UniqueName: \"kubernetes.io/projected/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-kube-api-access-rk9rj\") pod \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\" (UID: \"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e\") " Sep 15 12:07:02.806652 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.806594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw582\" (UniqueName: \"kubernetes.io/projected/cd7cd0ba-349a-48b6-9577-606aac4f8068-kube-api-access-xw582\") pod \"cd7cd0ba-349a-48b6-9577-606aac4f8068\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " Sep 15 12:07:02.806652 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.806633 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7cd0ba-349a-48b6-9577-606aac4f8068-openshift-service-ca\") pod \"cd7cd0ba-349a-48b6-9577-606aac4f8068\" (UID: \"cd7cd0ba-349a-48b6-9577-606aac4f8068\") " Sep 15 12:07:02.807097 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.806955 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e" (UID: "a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:07:02.807367 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.807281 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7cd0ba-349a-48b6-9577-606aac4f8068-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd7cd0ba-349a-48b6-9577-606aac4f8068" (UID: "cd7cd0ba-349a-48b6-9577-606aac4f8068"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:07:02.809419 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.809389 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cd7cd0ba-349a-48b6-9577-606aac4f8068" (UID: "cd7cd0ba-349a-48b6-9577-606aac4f8068"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:07:02.809554 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.809472 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd7cd0ba-349a-48b6-9577-606aac4f8068" (UID: "cd7cd0ba-349a-48b6-9577-606aac4f8068"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:07:02.809655 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.809625 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7cd0ba-349a-48b6-9577-606aac4f8068-kube-api-access-xw582" (OuterVolumeSpecName: "kube-api-access-xw582") pod "cd7cd0ba-349a-48b6-9577-606aac4f8068" (UID: "cd7cd0ba-349a-48b6-9577-606aac4f8068"). InnerVolumeSpecName "kube-api-access-xw582". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:07:02.809889 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.809866 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-kube-api-access-rk9rj" (OuterVolumeSpecName: "kube-api-access-rk9rj") pod "a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e" (UID: "a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e"). InnerVolumeSpecName "kube-api-access-rk9rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:07:02.908368 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908268 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct98f\" (UniqueName: \"kubernetes.io/projected/aa2daa27-f5c0-4458-bb7d-184d9e40a907-kube-api-access-ct98f\") pod \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\" (UID: \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\") " Sep 15 12:07:02.908368 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908311 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2daa27-f5c0-4458-bb7d-184d9e40a907-plugins-config-volume\") pod \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\" (UID: \"aa2daa27-f5c0-4458-bb7d-184d9e40a907\") " Sep 15 12:07:02.908600 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908589 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:07:02.908668 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908608 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd7cd0ba-349a-48b6-9577-606aac4f8068-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:07:02.908668 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908625 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:07:02.908668 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908640 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk9rj\" (UniqueName: \"kubernetes.io/projected/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e-kube-api-access-rk9rj\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:07:02.908668 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908655 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw582\" (UniqueName: \"kubernetes.io/projected/cd7cd0ba-349a-48b6-9577-606aac4f8068-kube-api-access-xw582\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:07:02.908887 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908668 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd7cd0ba-349a-48b6-9577-606aac4f8068-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:07:02.909008 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.908987 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2daa27-f5c0-4458-bb7d-184d9e40a907-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa2daa27-f5c0-4458-bb7d-184d9e40a907" (UID: "aa2daa27-f5c0-4458-bb7d-184d9e40a907"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:07:02.911148 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:02.911121 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2daa27-f5c0-4458-bb7d-184d9e40a907-kube-api-access-ct98f" (OuterVolumeSpecName: "kube-api-access-ct98f") pod "aa2daa27-f5c0-4458-bb7d-184d9e40a907" (UID: "aa2daa27-f5c0-4458-bb7d-184d9e40a907"). InnerVolumeSpecName "kube-api-access-ct98f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:07:03.009757 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.009712 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct98f\" (UniqueName: \"kubernetes.io/projected/aa2daa27-f5c0-4458-bb7d-184d9e40a907-kube-api-access-ct98f\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:07:03.009757 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.009745 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa2daa27-f5c0-4458-bb7d-184d9e40a907-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:07:03.658691 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.658608 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" Sep 15 12:07:03.659111 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.658605 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt" event={"ID":"aa2daa27-f5c0-4458-bb7d-184d9e40a907","Type":"ContainerDied","Data":"f78d3067d3ea07e5da317560ae79e80b1606454c8605462531611a991bba3a4e"} Sep 15 12:07:03.659111 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.658744 2579 scope.go:117] "RemoveContainer" containerID="e23eeddb6cf93d291ecb415bed10fb77adfe1c4f8ca67547a06e01dfa72a623a" Sep 15 12:07:03.660268 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.660240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" event={"ID":"a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e","Type":"ContainerDied","Data":"f113e949ad901cf537e37ceb2a5b3997883875401346429adc3eefffcda2aa18"} Sep 15 12:07:03.660379 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.660267 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm" Sep 15 12:07:03.662006 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.661984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" event={"ID":"cd7cd0ba-349a-48b6-9577-606aac4f8068","Type":"ContainerDied","Data":"f7c0cf9c3384fe5f6de60cae776d46e6cabae5dff52856c1f18d3839fa1781b1"} Sep 15 12:07:03.662128 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.661992 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf" Sep 15 12:07:03.669416 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.669391 2579 scope.go:117] "RemoveContainer" containerID="9d89637aa585c570b0a474ba181aee11d5923fe9ca844a331e012a1639245d2c" Sep 15 12:07:03.678585 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.678557 2579 scope.go:117] "RemoveContainer" containerID="cc51485b2ac446c89572fdac8b7722f7b342d51d27abcd2292cc84b29093d9e4" Sep 15 12:07:03.683185 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.683147 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt"] Sep 15 12:07:03.690255 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.690226 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-29d630-f755567f8-btsqt"] Sep 15 12:07:03.701727 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.701686 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf"] Sep 15 12:07:03.708010 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.707980 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-29d630-6cc77c5489-nxpnf"] Sep 15 12:07:03.718119 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.718086 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm"] Sep 15 12:07:03.722475 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:03.722446 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-29d630-7b77878895-bbrpm"] Sep 15 12:07:05.188352 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:05.188308 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e" path="/var/lib/kubelet/pods/a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e/volumes" Sep 15 12:07:05.188859 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:05.188832 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa2daa27-f5c0-4458-bb7d-184d9e40a907" path="/var/lib/kubelet/pods/aa2daa27-f5c0-4458-bb7d-184d9e40a907/volumes" Sep 15 12:07:05.189257 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:07:05.189238 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7cd0ba-349a-48b6-9577-606aac4f8068" path="/var/lib/kubelet/pods/cd7cd0ba-349a-48b6-9577-606aac4f8068/volumes" Sep 15 12:08:24.030303 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.030256 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq"] Sep 15 12:08:24.030896 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.030594 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" podUID="c20014d5-1a23-4a5d-b022-5a0b34192400" containerName="maas-api" containerID="cri-o://e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6" gracePeriod=30 Sep 15 12:08:24.102936 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.102900 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44"] Sep 15 12:08:24.103486 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.103454 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" podUID="1dec157b-20dc-49a1-ba71-f50cb28126b0" containerName="payload-processing" containerID="cri-o://1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7" gracePeriod=30 Sep 15 12:08:24.113231 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.112891 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm"] Sep 15 12:08:24.113346 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.113249 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" podUID="6b41998d-4972-420b-a6de-79f0807b46e6" containerName="payload-pre-processing" containerID="cri-o://2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f" gracePeriod=30 Sep 15 12:08:24.424637 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.424587 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 12:08:24.428702 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.428677 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 12:08:24.462031 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.461992 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b41998d-4972-420b-a6de-79f0807b46e6-plugins-config-volume\") pod \"6b41998d-4972-420b-a6de-79f0807b46e6\" (UID: \"6b41998d-4972-420b-a6de-79f0807b46e6\") " Sep 15 12:08:24.462259 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.462060 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec157b-20dc-49a1-ba71-f50cb28126b0-plugins-config-volume\") pod \"1dec157b-20dc-49a1-ba71-f50cb28126b0\" (UID: \"1dec157b-20dc-49a1-ba71-f50cb28126b0\") " Sep 15 12:08:24.462259 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.462144 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58lbf\" (UniqueName: \"kubernetes.io/projected/1dec157b-20dc-49a1-ba71-f50cb28126b0-kube-api-access-58lbf\") pod \"1dec157b-20dc-49a1-ba71-f50cb28126b0\" (UID: \"1dec157b-20dc-49a1-ba71-f50cb28126b0\") " Sep 15 12:08:24.462259 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.462169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7tt2\" (UniqueName: \"kubernetes.io/projected/6b41998d-4972-420b-a6de-79f0807b46e6-kube-api-access-t7tt2\") pod \"6b41998d-4972-420b-a6de-79f0807b46e6\" (UID: \"6b41998d-4972-420b-a6de-79f0807b46e6\") " Sep 15 12:08:24.462688 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.462641 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b41998d-4972-420b-a6de-79f0807b46e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b41998d-4972-420b-a6de-79f0807b46e6" (UID: "6b41998d-4972-420b-a6de-79f0807b46e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:08:24.463059 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.463026 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dec157b-20dc-49a1-ba71-f50cb28126b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dec157b-20dc-49a1-ba71-f50cb28126b0" (UID: "1dec157b-20dc-49a1-ba71-f50cb28126b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:08:24.467110 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.467072 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b41998d-4972-420b-a6de-79f0807b46e6-kube-api-access-t7tt2" (OuterVolumeSpecName: "kube-api-access-t7tt2") pod "6b41998d-4972-420b-a6de-79f0807b46e6" (UID: "6b41998d-4972-420b-a6de-79f0807b46e6"). InnerVolumeSpecName "kube-api-access-t7tt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:08:24.467635 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.467612 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dec157b-20dc-49a1-ba71-f50cb28126b0-kube-api-access-58lbf" (OuterVolumeSpecName: "kube-api-access-58lbf") pod "1dec157b-20dc-49a1-ba71-f50cb28126b0" (UID: "1dec157b-20dc-49a1-ba71-f50cb28126b0"). InnerVolumeSpecName "kube-api-access-58lbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:08:24.499267 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.499236 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 12:08:24.563635 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563542 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls\") pod \"c20014d5-1a23-4a5d-b022-5a0b34192400\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " Sep 15 12:08:24.563635 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563611 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c20014d5-1a23-4a5d-b022-5a0b34192400-openshift-service-ca\") pod \"c20014d5-1a23-4a5d-b022-5a0b34192400\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " Sep 15 12:08:24.563831 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-metrics-certs\") pod \"c20014d5-1a23-4a5d-b022-5a0b34192400\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " Sep 15 12:08:24.563831 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563693 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr2dg\" (UniqueName: \"kubernetes.io/projected/c20014d5-1a23-4a5d-b022-5a0b34192400-kube-api-access-mr2dg\") pod \"c20014d5-1a23-4a5d-b022-5a0b34192400\" (UID: \"c20014d5-1a23-4a5d-b022-5a0b34192400\") " Sep 15 12:08:24.563944 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563862 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lbf\" (UniqueName: \"kubernetes.io/projected/1dec157b-20dc-49a1-ba71-f50cb28126b0-kube-api-access-58lbf\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:24.563944 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563874 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7tt2\" (UniqueName: \"kubernetes.io/projected/6b41998d-4972-420b-a6de-79f0807b46e6-kube-api-access-t7tt2\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:24.563944 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563882 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b41998d-4972-420b-a6de-79f0807b46e6-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:24.563944 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563891 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dec157b-20dc-49a1-ba71-f50cb28126b0-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:24.563944 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.563924 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c20014d5-1a23-4a5d-b022-5a0b34192400-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c20014d5-1a23-4a5d-b022-5a0b34192400" (UID: "c20014d5-1a23-4a5d-b022-5a0b34192400"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:08:24.566020 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.565991 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c20014d5-1a23-4a5d-b022-5a0b34192400" (UID: "c20014d5-1a23-4a5d-b022-5a0b34192400"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:08:24.566020 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.566008 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c20014d5-1a23-4a5d-b022-5a0b34192400" (UID: "c20014d5-1a23-4a5d-b022-5a0b34192400"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:08:24.566020 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.566011 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20014d5-1a23-4a5d-b022-5a0b34192400-kube-api-access-mr2dg" (OuterVolumeSpecName: "kube-api-access-mr2dg") pod "c20014d5-1a23-4a5d-b022-5a0b34192400" (UID: "c20014d5-1a23-4a5d-b022-5a0b34192400"). InnerVolumeSpecName "kube-api-access-mr2dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:08:24.664906 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.664870 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:24.664906 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.664901 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr2dg\" (UniqueName: \"kubernetes.io/projected/c20014d5-1a23-4a5d-b022-5a0b34192400-kube-api-access-mr2dg\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:24.664906 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.664912 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c20014d5-1a23-4a5d-b022-5a0b34192400-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:24.665257 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:24.664922 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c20014d5-1a23-4a5d-b022-5a0b34192400-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:25.099055 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.099017 2579 generic.go:356] "Generic (PLEG): container finished" podID="c20014d5-1a23-4a5d-b022-5a0b34192400" containerID="e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6" exitCode=0 Sep 15 12:08:25.099564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.099067 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" Sep 15 12:08:25.099564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.099109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" event={"ID":"c20014d5-1a23-4a5d-b022-5a0b34192400","Type":"ContainerDied","Data":"e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6"} Sep 15 12:08:25.099564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.099168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq" event={"ID":"c20014d5-1a23-4a5d-b022-5a0b34192400","Type":"ContainerDied","Data":"61b267ad8d7540593d8ee3e5e675773023c3316d101fc671c9bb548901797d07"} Sep 15 12:08:25.099564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.099195 2579 scope.go:117] "RemoveContainer" containerID="e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6" Sep 15 12:08:25.101540 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.101514 2579 generic.go:356] "Generic (PLEG): container finished" podID="1dec157b-20dc-49a1-ba71-f50cb28126b0" containerID="1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7" exitCode=0 Sep 15 12:08:25.101686 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.101568 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" Sep 15 12:08:25.101756 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.101696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" event={"ID":"1dec157b-20dc-49a1-ba71-f50cb28126b0","Type":"ContainerDied","Data":"1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7"} Sep 15 12:08:25.101756 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.101737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44" event={"ID":"1dec157b-20dc-49a1-ba71-f50cb28126b0","Type":"ContainerDied","Data":"7e157b3414de60b5fd2493b6893a975b1d3f76fa7c8358a3fcc844e415b2435d"} Sep 15 12:08:25.105240 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.105215 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b41998d-4972-420b-a6de-79f0807b46e6" containerID="2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f" exitCode=0 Sep 15 12:08:25.105347 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.105266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" event={"ID":"6b41998d-4972-420b-a6de-79f0807b46e6","Type":"ContainerDied","Data":"2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f"} Sep 15 12:08:25.105347 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.105272 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" Sep 15 12:08:25.105347 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.105307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm" event={"ID":"6b41998d-4972-420b-a6de-79f0807b46e6","Type":"ContainerDied","Data":"f30c26fab88346b9c3a7016a7f70f870869d785df1df57d2255730db0e4bc35b"} Sep 15 12:08:25.110007 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.109989 2579 scope.go:117] "RemoveContainer" containerID="e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6" Sep 15 12:08:25.110463 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:08:25.110434 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6\": container with ID starting with e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6 not found: ID does not exist" containerID="e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6" Sep 15 12:08:25.110557 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.110473 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6"} err="failed to get container status \"e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6\": rpc error: code = NotFound desc = could not find container \"e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6\": container with ID starting with e4ef27c3b62f3ef910cf32ab1bbc9d75a217e0f145daacfcf325664515f493e6 not found: ID does not exist" Sep 15 12:08:25.110557 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.110497 2579 scope.go:117] "RemoveContainer" containerID="1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7" Sep 15 12:08:25.128494 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.128459 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44"] Sep 15 12:08:25.132961 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.132936 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ac76e5-67996c48f-hsk44"] Sep 15 12:08:25.135751 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.135733 2579 scope.go:117] "RemoveContainer" containerID="1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7" Sep 15 12:08:25.136136 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:08:25.136116 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7\": container with ID starting with 1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7 not found: ID does not exist" containerID="1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7" Sep 15 12:08:25.136339 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.136148 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7"} err="failed to get container status \"1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7\": rpc error: code = NotFound desc = could not find container \"1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7\": container with ID starting with 1ca52fe06f32a2a5520f8e19748876bceb22aa92e3681405512d4637342ad3e7 not found: ID does not exist" Sep 15 12:08:25.136339 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.136174 2579 scope.go:117] "RemoveContainer" containerID="2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f" Sep 15 12:08:25.143897 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.143873 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq"] Sep 15 12:08:25.145290 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.145269 2579 scope.go:117] "RemoveContainer" containerID="2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f" Sep 15 12:08:25.145656 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:08:25.145636 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f\": container with ID starting with 2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f not found: ID does not exist" containerID="2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f" Sep 15 12:08:25.145732 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.145663 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f"} err="failed to get container status \"2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f\": rpc error: code = NotFound desc = could not find container \"2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f\": container with ID starting with 2c8cf6a7f7022db3d5885427765efb0c359244a09920f1a16ff11d2e2475c87f not found: ID does not exist" Sep 15 12:08:25.148520 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.148498 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-ac76e5-557f76cf6d-z97vq"] Sep 15 12:08:25.161082 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.161053 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm"] Sep 15 12:08:25.165420 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.165393 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ac76e5-5dc6dd99f8-x4rmm"] Sep 15 12:08:25.188941 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.188913 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dec157b-20dc-49a1-ba71-f50cb28126b0" path="/var/lib/kubelet/pods/1dec157b-20dc-49a1-ba71-f50cb28126b0/volumes" Sep 15 12:08:25.189295 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.189282 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b41998d-4972-420b-a6de-79f0807b46e6" path="/var/lib/kubelet/pods/6b41998d-4972-420b-a6de-79f0807b46e6/volumes" Sep 15 12:08:25.189625 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:25.189613 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c20014d5-1a23-4a5d-b022-5a0b34192400" path="/var/lib/kubelet/pods/c20014d5-1a23-4a5d-b022-5a0b34192400/volumes" Sep 15 12:08:33.246981 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:33.246938 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:08:33.251070 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:33.251047 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:08:48.520344 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520292 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7881-e18cda"] Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520675 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b41998d-4972-420b-a6de-79f0807b46e6" containerName="payload-pre-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520687 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b41998d-4972-420b-a6de-79f0807b46e6" containerName="payload-pre-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520697 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" containerName="maas-api" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520702 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" containerName="maas-api" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520712 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da0b566f-95d3-4f6d-8a8c-339e356a3013" containerName="payload-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520718 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da0b566f-95d3-4f6d-8a8c-339e356a3013" containerName="payload-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520724 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d2797a9-934a-4be7-9fd5-76faf0cf3b35" containerName="maas-api" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520731 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d2797a9-934a-4be7-9fd5-76faf0cf3b35" containerName="maas-api" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520737 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebf51ba-2929-4c79-a662-f0822c10f76b" containerName="maas-api" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520742 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebf51ba-2929-4c79-a662-f0822c10f76b" containerName="maas-api" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520751 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dec157b-20dc-49a1-ba71-f50cb28126b0" containerName="payload-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520756 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dec157b-20dc-49a1-ba71-f50cb28126b0" containerName="payload-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520763 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa2daa27-f5c0-4458-bb7d-184d9e40a907" containerName="payload-pre-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520768 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa2daa27-f5c0-4458-bb7d-184d9e40a907" containerName="payload-pre-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520774 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e" containerName="payload-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520779 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e" containerName="payload-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520784 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56dfb2b-e546-487c-a86e-0cff0c657022" containerName="payload-pre-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520789 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56dfb2b-e546-487c-a86e-0cff0c657022" containerName="payload-pre-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520803 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fbe269-8c48-4364-9721-6a5db61f69ab" containerName="payload-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520807 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fbe269-8c48-4364-9721-6a5db61f69ab" containerName="payload-processing" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520814 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c20014d5-1a23-4a5d-b022-5a0b34192400" containerName="maas-api" Sep 15 12:08:48.520808 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520819 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c20014d5-1a23-4a5d-b022-5a0b34192400" containerName="maas-api" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520828 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7cd0ba-349a-48b6-9577-606aac4f8068" containerName="maas-api" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520834 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7cd0ba-349a-48b6-9577-606aac4f8068" containerName="maas-api" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520884 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7cd0ba-349a-48b6-9577-606aac4f8068" containerName="maas-api" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520892 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da0b566f-95d3-4f6d-8a8c-339e356a3013" containerName="payload-processing" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520897 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebf51ba-2929-4c79-a662-f0822c10f76b" containerName="maas-api" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520905 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c20014d5-1a23-4a5d-b022-5a0b34192400" containerName="maas-api" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520911 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7d1d26a-3430-4ee2-8135-f0fc5f4ec42e" containerName="payload-processing" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520917 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d2797a9-934a-4be7-9fd5-76faf0cf3b35" containerName="maas-api" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520923 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4cb7a8d-b7ab-48c3-b4b4-e420d50536c8" containerName="maas-api" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520928 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56dfb2b-e546-487c-a86e-0cff0c657022" containerName="payload-pre-processing" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520934 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b41998d-4972-420b-a6de-79f0807b46e6" containerName="payload-pre-processing" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520938 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa2daa27-f5c0-4458-bb7d-184d9e40a907" containerName="payload-pre-processing" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520943 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dec157b-20dc-49a1-ba71-f50cb28126b0" containerName="payload-processing" Sep 15 12:08:48.521616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.520948 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fbe269-8c48-4364-9721-6a5db61f69ab" containerName="payload-processing" Sep 15 12:08:48.523792 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.523777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-e18cda" Sep 15 12:08:48.526107 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.526086 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6twr\"" Sep 15 12:08:48.530753 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.530707 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-e18cda"] Sep 15 12:08:48.580147 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.580106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g56sx\" (UniqueName: \"kubernetes.io/projected/f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c-kube-api-access-g56sx\") pod \"test-curl-7881-e18cda\" (UID: \"f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c\") " pod="opendatahub/test-curl-7881-e18cda" Sep 15 12:08:48.680760 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.680717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g56sx\" (UniqueName: \"kubernetes.io/projected/f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c-kube-api-access-g56sx\") pod \"test-curl-7881-e18cda\" (UID: \"f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c\") " pod="opendatahub/test-curl-7881-e18cda" Sep 15 12:08:48.690376 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.690346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g56sx\" (UniqueName: \"kubernetes.io/projected/f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c-kube-api-access-g56sx\") pod \"test-curl-7881-e18cda\" (UID: \"f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c\") " pod="opendatahub/test-curl-7881-e18cda" Sep 15 12:08:48.833917 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:48.833824 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-e18cda" Sep 15 12:08:49.105270 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:49.105244 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-e18cda"] Sep 15 12:08:49.107252 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:08:49.107221 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7ce5b97_3a8d_4110_a1f6_e5d9b2572b0c.slice/crio-2511f1224e46fb43122d90c18af7c421c3a782a8e3b8fb2d6e60b8265a88ea55 WatchSource:0}: Error finding container 2511f1224e46fb43122d90c18af7c421c3a782a8e3b8fb2d6e60b8265a88ea55: Status 404 returned error can't find the container with id 2511f1224e46fb43122d90c18af7c421c3a782a8e3b8fb2d6e60b8265a88ea55 Sep 15 12:08:49.235592 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:49.235548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-e18cda" event={"ID":"f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c","Type":"ContainerStarted","Data":"a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd"} Sep 15 12:08:49.235592 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:49.235598 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-e18cda" event={"ID":"f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c","Type":"ContainerStarted","Data":"2511f1224e46fb43122d90c18af7c421c3a782a8e3b8fb2d6e60b8265a88ea55"} Sep 15 12:08:49.250816 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:49.250756 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7881-e18cda" podStartSLOduration=1.250739044 podStartE2EDuration="1.250739044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:08:49.250263526 +0000 UTC m=+1816.595372133" watchObservedRunningTime="2026-09-15 12:08:49.250739044 +0000 UTC m=+1816.595847649" Sep 15 12:08:49.866468 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:49.866432 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7881-e18cda"] Sep 15 12:08:49.870889 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:49.870854 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7881-e18cda"] Sep 15 12:08:50.241165 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:08:50.241130 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7881-e18cda" container="test-curl-7881-e18cda" Sep 15 12:08:51.245725 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:51.245677 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7881-e18cda" podUID="f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c" containerName="test-curl-7881-e18cda" containerID="cri-o://a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd" gracePeriod=2 Sep 15 12:08:53.436690 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:53.436662 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-e18cda" Sep 15 12:08:53.525113 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:53.525022 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g56sx\" (UniqueName: \"kubernetes.io/projected/f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c-kube-api-access-g56sx\") pod \"f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c\" (UID: \"f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c\") " Sep 15 12:08:53.527455 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:53.527427 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c-kube-api-access-g56sx" (OuterVolumeSpecName: "kube-api-access-g56sx") pod "f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c" (UID: "f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c"). InnerVolumeSpecName "kube-api-access-g56sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:08:53.625637 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:53.625604 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g56sx\" (UniqueName: \"kubernetes.io/projected/f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c-kube-api-access-g56sx\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:08:54.262617 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:54.262582 2579 generic.go:356] "Generic (PLEG): container finished" podID="f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c" containerID="a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd" exitCode=137 Sep 15 12:08:54.262777 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:54.262626 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-e18cda" Sep 15 12:08:54.262777 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:54.262682 2579 scope.go:117] "RemoveContainer" containerID="a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd" Sep 15 12:08:54.274663 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:54.274643 2579 scope.go:117] "RemoveContainer" containerID="a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd" Sep 15 12:08:54.275006 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:08:54.274984 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd\": container with ID starting with a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd not found: ID does not exist" containerID="a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd" Sep 15 12:08:54.275078 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:54.275015 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd"} err="failed to get container status \"a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd\": rpc error: code = NotFound desc = could not find container \"a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd\": container with ID starting with a6a5625daccf871efe8f09b8eb7cde35150beff988d8574747eec7462d6cb4dd not found: ID does not exist" Sep 15 12:08:54.275125 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:08:54.275076 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7881-e18cda\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c" pod="opendatahub/test-curl-7881-e18cda" Sep 15 12:08:55.188791 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:08:55.188753 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c" path="/var/lib/kubelet/pods/f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c/volumes" Sep 15 12:10:33.477846 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:33.477812 2579 scope.go:117] "RemoveContainer" containerID="0ae983b93b5543c1cb9e4026672df71d6c0b0e75bbd9dd2b579acc9e89a66af4" Sep 15 12:10:40.980452 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:40.980408 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7881-43e5e7"] Sep 15 12:10:40.980846 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:40.980800 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c" containerName="test-curl-7881-e18cda" Sep 15 12:10:40.980846 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:40.980811 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c" containerName="test-curl-7881-e18cda" Sep 15 12:10:40.980918 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:40.980874 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7ce5b97-3a8d-4110-a1f6-e5d9b2572b0c" containerName="test-curl-7881-e18cda" Sep 15 12:10:40.983996 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:40.983981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-43e5e7" Sep 15 12:10:40.987138 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:40.987120 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m6twr\"" Sep 15 12:10:40.996233 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:40.996184 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-43e5e7"] Sep 15 12:10:41.065892 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:41.065847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ld6\" (UniqueName: \"kubernetes.io/projected/a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1-kube-api-access-97ld6\") pod \"test-curl-7881-43e5e7\" (UID: \"a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1\") " pod="opendatahub/test-curl-7881-43e5e7" Sep 15 12:10:41.166716 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:41.166658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ld6\" (UniqueName: \"kubernetes.io/projected/a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1-kube-api-access-97ld6\") pod \"test-curl-7881-43e5e7\" (UID: \"a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1\") " pod="opendatahub/test-curl-7881-43e5e7" Sep 15 12:10:41.179990 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:41.179955 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ld6\" (UniqueName: \"kubernetes.io/projected/a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1-kube-api-access-97ld6\") pod \"test-curl-7881-43e5e7\" (UID: \"a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1\") " pod="opendatahub/test-curl-7881-43e5e7" Sep 15 12:10:41.293653 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:41.293554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-43e5e7" Sep 15 12:10:41.572415 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:41.572386 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-43e5e7"] Sep 15 12:10:41.574987 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:10:41.574960 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3e4c2fd_fecc_45f3_a6bb_9bed1508a0b1.slice/crio-f05b997d3d56f16a7ad9116f97816bac19df61def60ab6716a0324cc29fdf546 WatchSource:0}: Error finding container f05b997d3d56f16a7ad9116f97816bac19df61def60ab6716a0324cc29fdf546: Status 404 returned error can't find the container with id f05b997d3d56f16a7ad9116f97816bac19df61def60ab6716a0324cc29fdf546 Sep 15 12:10:41.807837 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:41.807797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-43e5e7" event={"ID":"a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1","Type":"ContainerStarted","Data":"30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7"} Sep 15 12:10:41.808021 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:41.807848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-43e5e7" event={"ID":"a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1","Type":"ContainerStarted","Data":"f05b997d3d56f16a7ad9116f97816bac19df61def60ab6716a0324cc29fdf546"} Sep 15 12:10:41.830059 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:41.829941 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7881-43e5e7" podStartSLOduration=1.8299239090000001 podStartE2EDuration="1.829923909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:41.828546378 +0000 UTC m=+1929.173654981" watchObservedRunningTime="2026-09-15 12:10:41.829923909 +0000 UTC m=+1929.175032578" Sep 15 12:10:42.472087 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.472052 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7881-43e5e7"] Sep 15 12:10:42.477246 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.477215 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7881-43e5e7"] Sep 15 12:10:42.611307 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.611227 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7881-f18bc4"] Sep 15 12:10:42.611615 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.611603 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1" containerName="test-curl-7881-43e5e7" Sep 15 12:10:42.611658 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.611618 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1" containerName="test-curl-7881-43e5e7" Sep 15 12:10:42.611693 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.611687 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1" containerName="test-curl-7881-43e5e7" Sep 15 12:10:42.614669 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.614651 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-f18bc4" Sep 15 12:10:42.621477 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.621448 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-f18bc4"] Sep 15 12:10:42.780992 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.780952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm5tp\" (UniqueName: \"kubernetes.io/projected/55ab1b68-b6cc-4f59-90a2-3c121ad6121c-kube-api-access-rm5tp\") pod \"test-curl-7881-f18bc4\" (UID: \"55ab1b68-b6cc-4f59-90a2-3c121ad6121c\") " pod="opendatahub/test-curl-7881-f18bc4" Sep 15 12:10:42.812814 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:10:42.812787 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7881-43e5e7" container="test-curl-7881-43e5e7" Sep 15 12:10:42.882474 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.882392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm5tp\" (UniqueName: \"kubernetes.io/projected/55ab1b68-b6cc-4f59-90a2-3c121ad6121c-kube-api-access-rm5tp\") pod \"test-curl-7881-f18bc4\" (UID: \"55ab1b68-b6cc-4f59-90a2-3c121ad6121c\") " pod="opendatahub/test-curl-7881-f18bc4" Sep 15 12:10:42.891558 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.891520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm5tp\" (UniqueName: \"kubernetes.io/projected/55ab1b68-b6cc-4f59-90a2-3c121ad6121c-kube-api-access-rm5tp\") pod \"test-curl-7881-f18bc4\" (UID: \"55ab1b68-b6cc-4f59-90a2-3c121ad6121c\") " pod="opendatahub/test-curl-7881-f18bc4" Sep 15 12:10:42.925531 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:42.925497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-f18bc4" Sep 15 12:10:43.199139 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:43.199115 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7881-f18bc4"] Sep 15 12:10:43.202121 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:10:43.202090 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ab1b68_b6cc_4f59_90a2_3c121ad6121c.slice/crio-2a79c8392e91121c8c89fc92f1551ab2541cb7ee7c8047f091c29047cf94e0e6 WatchSource:0}: Error finding container 2a79c8392e91121c8c89fc92f1551ab2541cb7ee7c8047f091c29047cf94e0e6: Status 404 returned error can't find the container with id 2a79c8392e91121c8c89fc92f1551ab2541cb7ee7c8047f091c29047cf94e0e6 Sep 15 12:10:43.819904 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:43.819866 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-f18bc4" event={"ID":"55ab1b68-b6cc-4f59-90a2-3c121ad6121c","Type":"ContainerStarted","Data":"f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d"} Sep 15 12:10:43.820398 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:43.819914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7881-f18bc4" event={"ID":"55ab1b68-b6cc-4f59-90a2-3c121ad6121c","Type":"ContainerStarted","Data":"2a79c8392e91121c8c89fc92f1551ab2541cb7ee7c8047f091c29047cf94e0e6"} Sep 15 12:10:43.820398 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:43.819935 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7881-43e5e7" podUID="a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1" containerName="test-curl-7881-43e5e7" containerID="cri-o://30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7" gracePeriod=2 Sep 15 12:10:43.836372 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:43.836313 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7881-f18bc4" podStartSLOduration=1.836295537 podStartE2EDuration="1.836295537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:10:43.834684382 +0000 UTC m=+1931.179792989" watchObservedRunningTime="2026-09-15 12:10:43.836295537 +0000 UTC m=+1931.181404195" Sep 15 12:10:44.450107 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:44.450069 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7881-f18bc4"] Sep 15 12:10:44.454616 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:44.454589 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7881-f18bc4"] Sep 15 12:10:44.824798 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:10:44.824766 2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7881-f18bc4" container="test-curl-7881-f18bc4" Sep 15 12:10:45.831292 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:45.831249 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7881-f18bc4" podUID="55ab1b68-b6cc-4f59-90a2-3c121ad6121c" containerName="test-curl-7881-f18bc4" containerID="cri-o://f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d" gracePeriod=2 Sep 15 12:10:45.999753 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:45.999725 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-43e5e7" Sep 15 12:10:46.111428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:46.111337 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97ld6\" (UniqueName: \"kubernetes.io/projected/a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1-kube-api-access-97ld6\") pod \"a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1\" (UID: \"a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1\") " Sep 15 12:10:46.113586 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:46.113563 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1-kube-api-access-97ld6" (OuterVolumeSpecName: "kube-api-access-97ld6") pod "a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1" (UID: "a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1"). InnerVolumeSpecName "kube-api-access-97ld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:10:46.213079 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:46.213042 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ld6\" (UniqueName: \"kubernetes.io/projected/a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1-kube-api-access-97ld6\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:10:46.837396 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:46.837362 2579 generic.go:356] "Generic (PLEG): container finished" podID="a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1" containerID="30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7" exitCode=137 Sep 15 12:10:46.837834 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:46.837407 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-43e5e7" Sep 15 12:10:46.837834 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:46.837438 2579 scope.go:117] "RemoveContainer" containerID="30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7" Sep 15 12:10:46.846555 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:46.846538 2579 scope.go:117] "RemoveContainer" containerID="30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7" Sep 15 12:10:46.846885 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:10:46.846859 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7\": container with ID starting with 30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7 not found: ID does not exist" containerID="30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7" Sep 15 12:10:46.846923 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:46.846896 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7"} err="failed to get container status \"30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7\": rpc error: code = NotFound desc = could not find container \"30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7\": container with ID starting with 30848d8d869abc3d9237a2cd1897e23daef1bf9d1ad2f191167cb74ee2a5baf7 not found: ID does not exist" Sep 15 12:10:46.846985 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:10:46.846964 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7881-43e5e7\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1" pod="opendatahub/test-curl-7881-43e5e7" Sep 15 12:10:47.187725 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:47.187692 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1" path="/var/lib/kubelet/pods/a3e4c2fd-fecc-45f3-a6bb-9bed1508a0b1/volumes" Sep 15 12:10:48.012764 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.012734 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-f18bc4" Sep 15 12:10:48.132259 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.132110 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm5tp\" (UniqueName: \"kubernetes.io/projected/55ab1b68-b6cc-4f59-90a2-3c121ad6121c-kube-api-access-rm5tp\") pod \"55ab1b68-b6cc-4f59-90a2-3c121ad6121c\" (UID: \"55ab1b68-b6cc-4f59-90a2-3c121ad6121c\") " Sep 15 12:10:48.134450 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.134418 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ab1b68-b6cc-4f59-90a2-3c121ad6121c-kube-api-access-rm5tp" (OuterVolumeSpecName: "kube-api-access-rm5tp") pod "55ab1b68-b6cc-4f59-90a2-3c121ad6121c" (UID: "55ab1b68-b6cc-4f59-90a2-3c121ad6121c"). InnerVolumeSpecName "kube-api-access-rm5tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:10:48.233569 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.233528 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm5tp\" (UniqueName: \"kubernetes.io/projected/55ab1b68-b6cc-4f59-90a2-3c121ad6121c-kube-api-access-rm5tp\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:10:48.849238 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.849179 2579 generic.go:356] "Generic (PLEG): container finished" podID="55ab1b68-b6cc-4f59-90a2-3c121ad6121c" containerID="f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d" exitCode=137 Sep 15 12:10:48.849449 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.849244 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7881-f18bc4" Sep 15 12:10:48.849449 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.849274 2579 scope.go:117] "RemoveContainer" containerID="f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d" Sep 15 12:10:48.858326 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.858310 2579 scope.go:117] "RemoveContainer" containerID="f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d" Sep 15 12:10:48.858586 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:10:48.858567 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d\": container with ID starting with f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d not found: ID does not exist" containerID="f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d" Sep 15 12:10:48.858628 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:48.858595 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d"} err="failed to get container status \"f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d\": rpc error: code = NotFound desc = could not find container \"f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d\": container with ID starting with f61218784b348a985dcf5db37ec7a6c7315b91f056a3476fbd33f56cf6ca726d not found: ID does not exist" Sep 15 12:10:48.859425 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:10:48.859404 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7881-f18bc4\" is forbidden: User \"system:node:ip-10-0-140-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-16.ec2.internal' and this object" podUID="55ab1b68-b6cc-4f59-90a2-3c121ad6121c" pod="opendatahub/test-curl-7881-f18bc4" Sep 15 12:10:49.187736 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:10:49.187708 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ab1b68-b6cc-4f59-90a2-3c121ad6121c" path="/var/lib/kubelet/pods/55ab1b68-b6cc-4f59-90a2-3c121ad6121c/volumes" Sep 15 12:12:13.053818 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.053773 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt"] Sep 15 12:12:13.054260 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.054195 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ab1b68-b6cc-4f59-90a2-3c121ad6121c" containerName="test-curl-7881-f18bc4" Sep 15 12:12:13.054260 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.054226 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ab1b68-b6cc-4f59-90a2-3c121ad6121c" containerName="test-curl-7881-f18bc4" Sep 15 12:12:13.054347 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.054301 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ab1b68-b6cc-4f59-90a2-3c121ad6121c" containerName="test-curl-7881-f18bc4" Sep 15 12:12:13.057620 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.057540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.061071 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.061043 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7139fb-serving-cert\"" Sep 15 12:12:13.068239 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.068176 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt"] Sep 15 12:12:13.175801 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.175766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.176025 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.175819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.176025 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.175907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.176025 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.175952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfq24\" (UniqueName: \"kubernetes.io/projected/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-kube-api-access-wfq24\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.277367 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.277333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.277564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.277377 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfq24\" (UniqueName: \"kubernetes.io/projected/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-kube-api-access-wfq24\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.277564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.277430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.277564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.277465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.277564 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:12:13.277503 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-serving-cert: secret "maas-api-e2e-ipp-a-7139fb-serving-cert" not found Sep 15 12:12:13.277717 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:12:13.277600 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls podName:66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b nodeName:}" failed. No retries permitted until 2026-09-15 12:12:13.777579974 +0000 UTC m=+2021.122688560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls") pod "maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" (UID: "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b") : secret "maas-api-e2e-ipp-a-7139fb-serving-cert" not found Sep 15 12:12:13.278036 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.278018 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.280092 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.280062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.298268 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.298181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfq24\" (UniqueName: \"kubernetes.io/projected/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-kube-api-access-wfq24\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.783261 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.783217 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.786251 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.786186 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:13.970511 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:13.970472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:14.069094 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.069012 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6"] Sep 15 12:12:14.072829 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.072806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:14.095760 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.095719 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7139fb\"" Sep 15 12:12:14.096284 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.096264 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7139fb-dockercfg-gv5jg\"" Sep 15 12:12:14.111877 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.111837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6"] Sep 15 12:12:14.139131 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.139092 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh"] Sep 15 12:12:14.142598 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.142570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:14.187621 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.187582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxgb\" (UniqueName: \"kubernetes.io/projected/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-kube-api-access-lfxgb\") pod \"payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh\" (UID: \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:14.187621 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.187624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz4jd\" (UniqueName: \"kubernetes.io/projected/59840da1-db12-46e6-9289-2a2fb7de7e2b-kube-api-access-vz4jd\") pod \"payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6\" (UID: \"59840da1-db12-46e6-9289-2a2fb7de7e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:14.187879 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.187648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh\" (UID: \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:14.187879 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.187773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59840da1-db12-46e6-9289-2a2fb7de7e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6\" (UID: \"59840da1-db12-46e6-9289-2a2fb7de7e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:14.191155 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.191124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh"] Sep 15 12:12:14.263849 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.263809 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt"] Sep 15 12:12:14.266339 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:12:14.266286 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fb8ac4_7b3a_4fe8_b34d_e5110bb34f6b.slice/crio-1ded8c9337fcbe6ed7327fc7809fed78c18a4062fcbaca23ae2cc6b3287f2236 WatchSource:0}: Error finding container 1ded8c9337fcbe6ed7327fc7809fed78c18a4062fcbaca23ae2cc6b3287f2236: Status 404 returned error can't find the container with id 1ded8c9337fcbe6ed7327fc7809fed78c18a4062fcbaca23ae2cc6b3287f2236 Sep 15 12:12:14.267973 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.267953 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:12:14.289090 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.289055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59840da1-db12-46e6-9289-2a2fb7de7e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6\" (UID: \"59840da1-db12-46e6-9289-2a2fb7de7e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:14.289306 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.289176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxgb\" (UniqueName: \"kubernetes.io/projected/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-kube-api-access-lfxgb\") pod \"payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh\" (UID: \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:14.289306 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.289231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz4jd\" (UniqueName: \"kubernetes.io/projected/59840da1-db12-46e6-9289-2a2fb7de7e2b-kube-api-access-vz4jd\") pod \"payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6\" (UID: \"59840da1-db12-46e6-9289-2a2fb7de7e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:14.289306 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.289256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh\" (UID: \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:14.289967 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.289942 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59840da1-db12-46e6-9289-2a2fb7de7e2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6\" (UID: \"59840da1-db12-46e6-9289-2a2fb7de7e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:14.290074 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.289986 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh\" (UID: \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:14.296104 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.296060 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" event={"ID":"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b","Type":"ContainerStarted","Data":"1ded8c9337fcbe6ed7327fc7809fed78c18a4062fcbaca23ae2cc6b3287f2236"} Sep 15 12:12:14.301241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.301189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxgb\" (UniqueName: \"kubernetes.io/projected/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-kube-api-access-lfxgb\") pod \"payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh\" (UID: \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:14.301382 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.301266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz4jd\" (UniqueName: \"kubernetes.io/projected/59840da1-db12-46e6-9289-2a2fb7de7e2b-kube-api-access-vz4jd\") pod \"payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6\" (UID: \"59840da1-db12-46e6-9289-2a2fb7de7e2b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:14.384920 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.384823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:14.453573 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.453545 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:14.663150 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.663124 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6"] Sep 15 12:12:14.665324 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:12:14.665299 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59840da1_db12_46e6_9289_2a2fb7de7e2b.slice/crio-3c538bdf8e2e9d0dde13c900d456be60710aea4a3037c981a8999992bbf2fb2d WatchSource:0}: Error finding container 3c538bdf8e2e9d0dde13c900d456be60710aea4a3037c981a8999992bbf2fb2d: Status 404 returned error can't find the container with id 3c538bdf8e2e9d0dde13c900d456be60710aea4a3037c981a8999992bbf2fb2d Sep 15 12:12:14.737454 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:14.737428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh"] Sep 15 12:12:14.739374 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:12:14.739348 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c6c366_72b2_40b8_ad2e_1d4b30a815b3.slice/crio-ff4ea71f6824167df5ce7585139bc61848f56a7c97273a778b66377371b23203 WatchSource:0}: Error finding container ff4ea71f6824167df5ce7585139bc61848f56a7c97273a778b66377371b23203: Status 404 returned error can't find the container with id ff4ea71f6824167df5ce7585139bc61848f56a7c97273a778b66377371b23203 Sep 15 12:12:15.312661 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:15.312610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" event={"ID":"76c6c366-72b2-40b8-ad2e-1d4b30a815b3","Type":"ContainerStarted","Data":"637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6"} Sep 15 12:12:15.313120 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:15.312716 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" event={"ID":"76c6c366-72b2-40b8-ad2e-1d4b30a815b3","Type":"ContainerStarted","Data":"ff4ea71f6824167df5ce7585139bc61848f56a7c97273a778b66377371b23203"} Sep 15 12:12:15.313120 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:15.312751 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:15.314435 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:15.314404 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" event={"ID":"59840da1-db12-46e6-9289-2a2fb7de7e2b","Type":"ContainerStarted","Data":"06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f"} Sep 15 12:12:15.314579 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:15.314464 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:15.314579 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:15.314490 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" event={"ID":"59840da1-db12-46e6-9289-2a2fb7de7e2b","Type":"ContainerStarted","Data":"3c538bdf8e2e9d0dde13c900d456be60710aea4a3037c981a8999992bbf2fb2d"} Sep 15 12:12:15.330269 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:15.330214 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" podStartSLOduration=1.330180129 podStartE2EDuration="1.330180129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:15.328698644 +0000 UTC m=+2022.673807249" watchObservedRunningTime="2026-09-15 12:12:15.330180129 +0000 UTC m=+2022.675288738" Sep 15 12:12:15.346961 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:15.346895 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" podStartSLOduration=1.34687367 podStartE2EDuration="1.34687367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:15.344397665 +0000 UTC m=+2022.689506273" watchObservedRunningTime="2026-09-15 12:12:15.34687367 +0000 UTC m=+2022.691982281" Sep 15 12:12:17.326880 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:17.326790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" event={"ID":"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b","Type":"ContainerStarted","Data":"c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a"} Sep 15 12:12:17.326880 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:17.326856 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:17.346913 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:17.346818 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" podStartSLOduration=1.5609447730000001 podStartE2EDuration="4.346793048s" totalImagesPullingTime="2.785848275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:17.345972341 +0000 UTC m=+2024.691080961" watchObservedRunningTime="2026-09-15 12:12:17.346793048 +0000 UTC m=+2024.691901655" Sep 15 12:12:23.337742 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:23.337712 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:12:26.322355 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:26.322321 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:12:26.322813 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:26.322421 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:12:43.265988 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.265936 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw"] Sep 15 12:12:43.269557 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.269534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.271767 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.271744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-74aca3-serving-cert\"" Sep 15 12:12:43.280176 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.280144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw"] Sep 15 12:12:43.359369 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.359327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bae194-0323-440a-928e-3355bc1ce80f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.359369 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.359376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg77n\" (UniqueName: \"kubernetes.io/projected/57bae194-0323-440a-928e-3355bc1ce80f-kube-api-access-mg77n\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.359610 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.359404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-metrics-certs\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.359610 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.359446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.460536 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.460496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg77n\" (UniqueName: \"kubernetes.io/projected/57bae194-0323-440a-928e-3355bc1ce80f-kube-api-access-mg77n\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.460536 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.460538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-metrics-certs\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.460769 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.460582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.460769 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.460644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bae194-0323-440a-928e-3355bc1ce80f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.461270 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.461239 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bae194-0323-440a-928e-3355bc1ce80f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.463371 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.463349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-metrics-certs\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.463479 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.463386 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.470030 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.470006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg77n\" (UniqueName: \"kubernetes.io/projected/57bae194-0323-440a-928e-3355bc1ce80f-kube-api-access-mg77n\") pod \"maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.582913 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.582819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:43.864048 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:43.864014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw"] Sep 15 12:12:43.866065 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:12:43.866028 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bae194_0323_440a_928e_3355bc1ce80f.slice/crio-aee8a1eddff849cb474393a418648792a5e5f2115b50cde9d0038156bc35664f WatchSource:0}: Error finding container aee8a1eddff849cb474393a418648792a5e5f2115b50cde9d0038156bc35664f: Status 404 returned error can't find the container with id aee8a1eddff849cb474393a418648792a5e5f2115b50cde9d0038156bc35664f Sep 15 12:12:44.481519 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:44.481479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" event={"ID":"57bae194-0323-440a-928e-3355bc1ce80f","Type":"ContainerStarted","Data":"0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d"} Sep 15 12:12:44.481992 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:44.481532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" event={"ID":"57bae194-0323-440a-928e-3355bc1ce80f","Type":"ContainerStarted","Data":"aee8a1eddff849cb474393a418648792a5e5f2115b50cde9d0038156bc35664f"} Sep 15 12:12:44.481992 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:44.481568 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:12:44.504382 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:44.504273 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" podStartSLOduration=1.135367775 podStartE2EDuration="1.504253176s" totalImagesPullingTime="368.885401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:12:44.502624874 +0000 UTC m=+2051.847733503" watchObservedRunningTime="2026-09-15 12:12:44.504253176 +0000 UTC m=+2051.849361782" Sep 15 12:12:50.491717 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:12:50.491688 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:13:33.280607 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:13:33.280574 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:13:33.285909 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:13:33.285884 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:13:53.525527 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:13:53.525494 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh_76c6c366-72b2-40b8-ad2e-1d4b30a815b3/payload-processing/0.log" Sep 15 12:14:02.768684 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.768642 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k"] Sep 15 12:14:02.772370 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.772349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.777240 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.777190 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-5fd646-serving-cert\"" Sep 15 12:14:02.789436 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.789403 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k"] Sep 15 12:14:02.876172 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.876133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxrl\" (UniqueName: \"kubernetes.io/projected/57e2a905-94ce-4df8-bc53-8bb681e64b18-kube-api-access-qfxrl\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.876428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.876290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2a905-94ce-4df8-bc53-8bb681e64b18-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.876428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.876320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.876428 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.876351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.977479 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.977438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2a905-94ce-4df8-bc53-8bb681e64b18-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.977479 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.977485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.977732 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.977515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.977732 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.977569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxrl\" (UniqueName: \"kubernetes.io/projected/57e2a905-94ce-4df8-bc53-8bb681e64b18-kube-api-access-qfxrl\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.977732 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:14:02.977677 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-serving-cert: secret "maas-api-e2e-ipp-cleanup-5fd646-serving-cert" not found Sep 15 12:14:02.977858 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:14:02.977774 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls podName:57e2a905-94ce-4df8-bc53-8bb681e64b18 nodeName:}" failed. No retries permitted until 2026-09-15 12:14:03.477750442 +0000 UTC m=+2130.822859042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" (UID: "57e2a905-94ce-4df8-bc53-8bb681e64b18") : secret "maas-api-e2e-ipp-cleanup-5fd646-serving-cert" not found Sep 15 12:14:02.978165 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.978139 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2a905-94ce-4df8-bc53-8bb681e64b18-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.980167 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.980148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:02.988286 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:02.988257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxrl\" (UniqueName: \"kubernetes.io/projected/57e2a905-94ce-4df8-bc53-8bb681e64b18-kube-api-access-qfxrl\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:03.483037 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.483002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:03.485720 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.485690 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:03.682690 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.682647 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5"] Sep 15 12:14:03.683586 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.683565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:03.686990 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.686880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:03.690530 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.690505 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5fd646\"" Sep 15 12:14:03.704921 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.704876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5"] Sep 15 12:14:03.728697 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.728658 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6"] Sep 15 12:14:03.733511 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.733435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:03.736277 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.736251 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5fd646-dockercfg-997bp\"" Sep 15 12:14:03.744063 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.744029 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6"] Sep 15 12:14:03.785546 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.785506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/2877f62e-7c68-4c76-a728-d4005a4c6bab-kube-api-access-rf4hb\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5\" (UID: \"2877f62e-7c68-4c76-a728-d4005a4c6bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:03.785909 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.785628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmzz\" (UniqueName: \"kubernetes.io/projected/87e88d09-f545-474f-a4b0-77d7a651aa5f-kube-api-access-fhmzz\") pod \"payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6\" (UID: \"87e88d09-f545-474f-a4b0-77d7a651aa5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:03.785909 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.785661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877f62e-7c68-4c76-a728-d4005a4c6bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5\" (UID: \"2877f62e-7c68-4c76-a728-d4005a4c6bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:03.785909 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.785726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e88d09-f545-474f-a4b0-77d7a651aa5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6\" (UID: \"87e88d09-f545-474f-a4b0-77d7a651aa5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:03.886885 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.886846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/2877f62e-7c68-4c76-a728-d4005a4c6bab-kube-api-access-rf4hb\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5\" (UID: \"2877f62e-7c68-4c76-a728-d4005a4c6bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:03.887088 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.886907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmzz\" (UniqueName: \"kubernetes.io/projected/87e88d09-f545-474f-a4b0-77d7a651aa5f-kube-api-access-fhmzz\") pod \"payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6\" (UID: \"87e88d09-f545-474f-a4b0-77d7a651aa5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:03.887088 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.886925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877f62e-7c68-4c76-a728-d4005a4c6bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5\" (UID: \"2877f62e-7c68-4c76-a728-d4005a4c6bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:03.887088 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.886962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e88d09-f545-474f-a4b0-77d7a651aa5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6\" (UID: \"87e88d09-f545-474f-a4b0-77d7a651aa5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:03.887775 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.887744 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e88d09-f545-474f-a4b0-77d7a651aa5f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6\" (UID: \"87e88d09-f545-474f-a4b0-77d7a651aa5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:03.887886 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.887789 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877f62e-7c68-4c76-a728-d4005a4c6bab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5\" (UID: \"2877f62e-7c68-4c76-a728-d4005a4c6bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:03.896752 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.896727 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/2877f62e-7c68-4c76-a728-d4005a4c6bab-kube-api-access-rf4hb\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5\" (UID: \"2877f62e-7c68-4c76-a728-d4005a4c6bab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:03.896858 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.896764 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmzz\" (UniqueName: \"kubernetes.io/projected/87e88d09-f545-474f-a4b0-77d7a651aa5f-kube-api-access-fhmzz\") pod \"payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6\" (UID: \"87e88d09-f545-474f-a4b0-77d7a651aa5f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:03.978422 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:03.978398 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k"] Sep 15 12:14:03.980257 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:14:03.980191 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e2a905_94ce_4df8_bc53_8bb681e64b18.slice/crio-77ec160e5672705febd055e5f9d363acaa7bf341592b78a39620a5ecb9808202 WatchSource:0}: Error finding container 77ec160e5672705febd055e5f9d363acaa7bf341592b78a39620a5ecb9808202: Status 404 returned error can't find the container with id 77ec160e5672705febd055e5f9d363acaa7bf341592b78a39620a5ecb9808202 Sep 15 12:14:04.020174 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.020122 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:04.048348 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.048316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:04.299805 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.298889 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5"] Sep 15 12:14:04.302327 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:14:04.302292 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2877f62e_7c68_4c76_a728_d4005a4c6bab.slice/crio-0993002eb91765f9258ada07c3b5fb51c583b6ba57827ff383cfcda09de13226 WatchSource:0}: Error finding container 0993002eb91765f9258ada07c3b5fb51c583b6ba57827ff383cfcda09de13226: Status 404 returned error can't find the container with id 0993002eb91765f9258ada07c3b5fb51c583b6ba57827ff383cfcda09de13226 Sep 15 12:14:04.333109 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.333083 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6"] Sep 15 12:14:04.339090 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:14:04.335874 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e88d09_f545_474f_a4b0_77d7a651aa5f.slice/crio-2ea1fed81ae8df456a0d39c86306e018dd0d3b55664855887491e11dac7cf807 WatchSource:0}: Error finding container 2ea1fed81ae8df456a0d39c86306e018dd0d3b55664855887491e11dac7cf807: Status 404 returned error can't find the container with id 2ea1fed81ae8df456a0d39c86306e018dd0d3b55664855887491e11dac7cf807 Sep 15 12:14:04.892838 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.892804 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" event={"ID":"2877f62e-7c68-4c76-a728-d4005a4c6bab","Type":"ContainerStarted","Data":"f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1"} Sep 15 12:14:04.893367 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.892867 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" event={"ID":"2877f62e-7c68-4c76-a728-d4005a4c6bab","Type":"ContainerStarted","Data":"0993002eb91765f9258ada07c3b5fb51c583b6ba57827ff383cfcda09de13226"} Sep 15 12:14:04.893367 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.892898 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:04.894601 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.894574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" event={"ID":"87e88d09-f545-474f-a4b0-77d7a651aa5f","Type":"ContainerStarted","Data":"eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b"} Sep 15 12:14:04.894711 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.894636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" event={"ID":"87e88d09-f545-474f-a4b0-77d7a651aa5f","Type":"ContainerStarted","Data":"2ea1fed81ae8df456a0d39c86306e018dd0d3b55664855887491e11dac7cf807"} Sep 15 12:14:04.894711 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.894662 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:04.896126 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.896102 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" event={"ID":"57e2a905-94ce-4df8-bc53-8bb681e64b18","Type":"ContainerStarted","Data":"8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede"} Sep 15 12:14:04.896264 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.896142 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" event={"ID":"57e2a905-94ce-4df8-bc53-8bb681e64b18","Type":"ContainerStarted","Data":"77ec160e5672705febd055e5f9d363acaa7bf341592b78a39620a5ecb9808202"} Sep 15 12:14:04.911332 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.911280 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" podStartSLOduration=1.911263647 podStartE2EDuration="1.911263647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:14:04.909846256 +0000 UTC m=+2132.254954863" watchObservedRunningTime="2026-09-15 12:14:04.911263647 +0000 UTC m=+2132.256372262" Sep 15 12:14:04.929577 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.929520 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" podStartSLOduration=2.580597621 podStartE2EDuration="2.929505028s" totalImagesPullingTime="348.907407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:14:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:14:04.92843353 +0000 UTC m=+2132.273542135" watchObservedRunningTime="2026-09-15 12:14:04.929505028 +0000 UTC m=+2132.274613630" Sep 15 12:14:04.946932 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:04.946862 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" podStartSLOduration=1.94683735 podStartE2EDuration="1.94683735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:14:04.944485608 +0000 UTC m=+2132.289594214" watchObservedRunningTime="2026-09-15 12:14:04.94683735 +0000 UTC m=+2132.291945963" Sep 15 12:14:05.903964 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:05.903931 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:11.914393 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:11.914360 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:15.904974 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:15.904943 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:15.905392 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:15.905015 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:40.523716 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.523619 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k"] Sep 15 12:14:40.524299 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.523876 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" podUID="57e2a905-94ce-4df8-bc53-8bb681e64b18" containerName="maas-api" containerID="cri-o://8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede" gracePeriod=30 Sep 15 12:14:40.582788 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.582754 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6"] Sep 15 12:14:40.582985 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.582955 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" podUID="87e88d09-f545-474f-a4b0-77d7a651aa5f" containerName="payload-processing" containerID="cri-o://eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b" gracePeriod=30 Sep 15 12:14:40.592824 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.592793 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5"] Sep 15 12:14:40.593088 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.593044 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" podUID="2877f62e-7c68-4c76-a728-d4005a4c6bab" containerName="payload-pre-processing" containerID="cri-o://f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1" gracePeriod=30 Sep 15 12:14:40.875673 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.875645 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:40.895565 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.895539 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:40.903358 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.903331 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:40.918838 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.918789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877f62e-7c68-4c76-a728-d4005a4c6bab-plugins-config-volume\") pod \"2877f62e-7c68-4c76-a728-d4005a4c6bab\" (UID: \"2877f62e-7c68-4c76-a728-d4005a4c6bab\") " Sep 15 12:14:40.919024 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.918855 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e88d09-f545-474f-a4b0-77d7a651aa5f-plugins-config-volume\") pod \"87e88d09-f545-474f-a4b0-77d7a651aa5f\" (UID: \"87e88d09-f545-474f-a4b0-77d7a651aa5f\") " Sep 15 12:14:40.919024 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.918894 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2a905-94ce-4df8-bc53-8bb681e64b18-openshift-service-ca\") pod \"57e2a905-94ce-4df8-bc53-8bb681e64b18\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " Sep 15 12:14:40.919024 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.918921 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-metrics-certs\") pod \"57e2a905-94ce-4df8-bc53-8bb681e64b18\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " Sep 15 12:14:40.919024 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.918999 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls\") pod \"57e2a905-94ce-4df8-bc53-8bb681e64b18\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " Sep 15 12:14:40.919348 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.919062 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfxrl\" (UniqueName: \"kubernetes.io/projected/57e2a905-94ce-4df8-bc53-8bb681e64b18-kube-api-access-qfxrl\") pod \"57e2a905-94ce-4df8-bc53-8bb681e64b18\" (UID: \"57e2a905-94ce-4df8-bc53-8bb681e64b18\") " Sep 15 12:14:40.919348 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.919098 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/2877f62e-7c68-4c76-a728-d4005a4c6bab-kube-api-access-rf4hb\") pod \"2877f62e-7c68-4c76-a728-d4005a4c6bab\" (UID: \"2877f62e-7c68-4c76-a728-d4005a4c6bab\") " Sep 15 12:14:40.919348 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.919128 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhmzz\" (UniqueName: \"kubernetes.io/projected/87e88d09-f545-474f-a4b0-77d7a651aa5f-kube-api-access-fhmzz\") pod \"87e88d09-f545-474f-a4b0-77d7a651aa5f\" (UID: \"87e88d09-f545-474f-a4b0-77d7a651aa5f\") " Sep 15 12:14:40.919505 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.919458 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2877f62e-7c68-4c76-a728-d4005a4c6bab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2877f62e-7c68-4c76-a728-d4005a4c6bab" (UID: "2877f62e-7c68-4c76-a728-d4005a4c6bab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:14:40.921784 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.919726 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e88d09-f545-474f-a4b0-77d7a651aa5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87e88d09-f545-474f-a4b0-77d7a651aa5f" (UID: "87e88d09-f545-474f-a4b0-77d7a651aa5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:14:40.921784 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.919858 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e2a905-94ce-4df8-bc53-8bb681e64b18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57e2a905-94ce-4df8-bc53-8bb681e64b18" (UID: "57e2a905-94ce-4df8-bc53-8bb681e64b18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:14:40.923252 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.923164 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e88d09-f545-474f-a4b0-77d7a651aa5f-kube-api-access-fhmzz" (OuterVolumeSpecName: "kube-api-access-fhmzz") pod "87e88d09-f545-474f-a4b0-77d7a651aa5f" (UID: "87e88d09-f545-474f-a4b0-77d7a651aa5f"). InnerVolumeSpecName "kube-api-access-fhmzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:14:40.926845 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.926804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2877f62e-7c68-4c76-a728-d4005a4c6bab-kube-api-access-rf4hb" (OuterVolumeSpecName: "kube-api-access-rf4hb") pod "2877f62e-7c68-4c76-a728-d4005a4c6bab" (UID: "2877f62e-7c68-4c76-a728-d4005a4c6bab"). InnerVolumeSpecName "kube-api-access-rf4hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:14:40.926845 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.926835 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57e2a905-94ce-4df8-bc53-8bb681e64b18" (UID: "57e2a905-94ce-4df8-bc53-8bb681e64b18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:14:40.927011 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.926910 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57e2a905-94ce-4df8-bc53-8bb681e64b18" (UID: "57e2a905-94ce-4df8-bc53-8bb681e64b18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:14:40.928394 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:40.928367 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e2a905-94ce-4df8-bc53-8bb681e64b18-kube-api-access-qfxrl" (OuterVolumeSpecName: "kube-api-access-qfxrl") pod "57e2a905-94ce-4df8-bc53-8bb681e64b18" (UID: "57e2a905-94ce-4df8-bc53-8bb681e64b18"). InnerVolumeSpecName "kube-api-access-qfxrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:14:41.020894 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.020842 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxrl\" (UniqueName: \"kubernetes.io/projected/57e2a905-94ce-4df8-bc53-8bb681e64b18-kube-api-access-qfxrl\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:14:41.020894 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.020893 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf4hb\" (UniqueName: \"kubernetes.io/projected/2877f62e-7c68-4c76-a728-d4005a4c6bab-kube-api-access-rf4hb\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:14:41.021397 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.020906 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhmzz\" (UniqueName: \"kubernetes.io/projected/87e88d09-f545-474f-a4b0-77d7a651aa5f-kube-api-access-fhmzz\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:14:41.021397 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.020920 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2877f62e-7c68-4c76-a728-d4005a4c6bab-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:14:41.021397 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.020933 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e88d09-f545-474f-a4b0-77d7a651aa5f-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:14:41.021397 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.020946 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e2a905-94ce-4df8-bc53-8bb681e64b18-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:14:41.021397 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.020960 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:14:41.021397 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.020971 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57e2a905-94ce-4df8-bc53-8bb681e64b18-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:14:41.096091 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.096002 2579 generic.go:356] "Generic (PLEG): container finished" podID="2877f62e-7c68-4c76-a728-d4005a4c6bab" containerID="f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1" exitCode=0 Sep 15 12:14:41.096091 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.096049 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" Sep 15 12:14:41.096404 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.096094 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" event={"ID":"2877f62e-7c68-4c76-a728-d4005a4c6bab","Type":"ContainerDied","Data":"f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1"} Sep 15 12:14:41.096404 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.096150 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5" event={"ID":"2877f62e-7c68-4c76-a728-d4005a4c6bab","Type":"ContainerDied","Data":"0993002eb91765f9258ada07c3b5fb51c583b6ba57827ff383cfcda09de13226"} Sep 15 12:14:41.096404 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.096178 2579 scope.go:117] "RemoveContainer" containerID="f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1" Sep 15 12:14:41.097526 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.097504 2579 generic.go:356] "Generic (PLEG): container finished" podID="87e88d09-f545-474f-a4b0-77d7a651aa5f" containerID="eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b" exitCode=0 Sep 15 12:14:41.097615 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.097556 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" Sep 15 12:14:41.097615 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.097595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" event={"ID":"87e88d09-f545-474f-a4b0-77d7a651aa5f","Type":"ContainerDied","Data":"eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b"} Sep 15 12:14:41.097711 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.097648 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6" event={"ID":"87e88d09-f545-474f-a4b0-77d7a651aa5f","Type":"ContainerDied","Data":"2ea1fed81ae8df456a0d39c86306e018dd0d3b55664855887491e11dac7cf807"} Sep 15 12:14:41.099115 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.099095 2579 generic.go:356] "Generic (PLEG): container finished" podID="57e2a905-94ce-4df8-bc53-8bb681e64b18" containerID="8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede" exitCode=0 Sep 15 12:14:41.099353 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.099332 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" event={"ID":"57e2a905-94ce-4df8-bc53-8bb681e64b18","Type":"ContainerDied","Data":"8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede"} Sep 15 12:14:41.099494 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.099480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" event={"ID":"57e2a905-94ce-4df8-bc53-8bb681e64b18","Type":"ContainerDied","Data":"77ec160e5672705febd055e5f9d363acaa7bf341592b78a39620a5ecb9808202"} Sep 15 12:14:41.099641 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.099629 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k" Sep 15 12:14:41.111010 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.110991 2579 scope.go:117] "RemoveContainer" containerID="f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1" Sep 15 12:14:41.111540 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:14:41.111509 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1\": container with ID starting with f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1 not found: ID does not exist" containerID="f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1" Sep 15 12:14:41.111940 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.111551 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1"} err="failed to get container status \"f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1\": rpc error: code = NotFound desc = could not find container \"f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1\": container with ID starting with f6479317411a019eb4268d058257e78bb0916dc0d41261605b30697486ccb7c1 not found: ID does not exist" Sep 15 12:14:41.111940 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.111573 2579 scope.go:117] "RemoveContainer" containerID="eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b" Sep 15 12:14:41.121266 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.121243 2579 scope.go:117] "RemoveContainer" containerID="eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b" Sep 15 12:14:41.121649 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:14:41.121627 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b\": container with ID starting with eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b not found: ID does not exist" containerID="eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b" Sep 15 12:14:41.121731 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.121655 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b"} err="failed to get container status \"eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b\": rpc error: code = NotFound desc = could not find container \"eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b\": container with ID starting with eb5fc59842510a0ee03a7e7b58c2419f6460643c0a2dcd4568123c0732b5e55b not found: ID does not exist" Sep 15 12:14:41.121731 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.121674 2579 scope.go:117] "RemoveContainer" containerID="8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede" Sep 15 12:14:41.128881 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.128848 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k"] Sep 15 12:14:41.131517 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.131484 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-5fd646-775c8d4f68-txv5k"] Sep 15 12:14:41.133327 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.133303 2579 scope.go:117] "RemoveContainer" containerID="8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede" Sep 15 12:14:41.133851 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:14:41.133829 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede\": container with ID starting with 8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede not found: ID does not exist" containerID="8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede" Sep 15 12:14:41.133953 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.133858 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede"} err="failed to get container status \"8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede\": rpc error: code = NotFound desc = could not find container \"8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede\": container with ID starting with 8e7959c51e86354a519d6869c3bb6f043a27cb5ce6b09138d7f0f0752d133ede not found: ID does not exist" Sep 15 12:14:41.146441 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.146413 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5"] Sep 15 12:14:41.154484 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.154455 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fd646-65f74fbb54-tpzt5"] Sep 15 12:14:41.165820 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.165784 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6"] Sep 15 12:14:41.167699 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.167679 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5fd646-6d954f869-r9qn6"] Sep 15 12:14:41.188268 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.188238 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2877f62e-7c68-4c76-a728-d4005a4c6bab" path="/var/lib/kubelet/pods/2877f62e-7c68-4c76-a728-d4005a4c6bab/volumes" Sep 15 12:14:41.188644 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.188631 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e2a905-94ce-4df8-bc53-8bb681e64b18" path="/var/lib/kubelet/pods/57e2a905-94ce-4df8-bc53-8bb681e64b18/volumes" Sep 15 12:14:41.188942 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:14:41.188931 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87e88d09-f545-474f-a4b0-77d7a651aa5f" path="/var/lib/kubelet/pods/87e88d09-f545-474f-a4b0-77d7a651aa5f/volumes" Sep 15 12:15:07.743995 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:07.743961 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt"] Sep 15 12:15:07.744430 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:07.744214 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" podUID="66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" containerName="maas-api" containerID="cri-o://c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a" gracePeriod=30 Sep 15 12:15:07.804973 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:07.804938 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh"] Sep 15 12:15:07.805378 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:07.805179 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" podUID="76c6c366-72b2-40b8-ad2e-1d4b30a815b3" containerName="payload-processing" containerID="cri-o://637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6" gracePeriod=30 Sep 15 12:15:07.820885 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:07.820851 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6"] Sep 15 12:15:07.821080 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:07.821059 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" podUID="59840da1-db12-46e6-9289-2a2fb7de7e2b" containerName="payload-pre-processing" containerID="cri-o://06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f" gracePeriod=30 Sep 15 12:15:08.113874 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.113848 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:15:08.143655 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.143630 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:15:08.147457 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.147434 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:15:08.251096 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.250993 2579 generic.go:356] "Generic (PLEG): container finished" podID="76c6c366-72b2-40b8-ad2e-1d4b30a815b3" containerID="637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6" exitCode=0 Sep 15 12:15:08.251096 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.251037 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" Sep 15 12:15:08.251096 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.251080 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" event={"ID":"76c6c366-72b2-40b8-ad2e-1d4b30a815b3","Type":"ContainerDied","Data":"637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6"} Sep 15 12:15:08.251395 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.251143 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh" event={"ID":"76c6c366-72b2-40b8-ad2e-1d4b30a815b3","Type":"ContainerDied","Data":"ff4ea71f6824167df5ce7585139bc61848f56a7c97273a778b66377371b23203"} Sep 15 12:15:08.251395 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.251173 2579 scope.go:117] "RemoveContainer" containerID="637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6" Sep 15 12:15:08.252564 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.252545 2579 generic.go:356] "Generic (PLEG): container finished" podID="59840da1-db12-46e6-9289-2a2fb7de7e2b" containerID="06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f" exitCode=0 Sep 15 12:15:08.252680 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.252578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" event={"ID":"59840da1-db12-46e6-9289-2a2fb7de7e2b","Type":"ContainerDied","Data":"06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f"} Sep 15 12:15:08.252680 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.252599 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" Sep 15 12:15:08.252680 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.252629 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6" event={"ID":"59840da1-db12-46e6-9289-2a2fb7de7e2b","Type":"ContainerDied","Data":"3c538bdf8e2e9d0dde13c900d456be60710aea4a3037c981a8999992bbf2fb2d"} Sep 15 12:15:08.254035 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.254013 2579 generic.go:356] "Generic (PLEG): container finished" podID="66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" containerID="c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a" exitCode=0 Sep 15 12:15:08.254137 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.254057 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" Sep 15 12:15:08.254137 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.254084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" event={"ID":"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b","Type":"ContainerDied","Data":"c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a"} Sep 15 12:15:08.254137 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.254123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt" event={"ID":"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b","Type":"ContainerDied","Data":"1ded8c9337fcbe6ed7327fc7809fed78c18a4062fcbaca23ae2cc6b3287f2236"} Sep 15 12:15:08.260358 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.260338 2579 scope.go:117] "RemoveContainer" containerID="637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6" Sep 15 12:15:08.260674 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:15:08.260637 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6\": container with ID starting with 637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6 not found: ID does not exist" containerID="637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6" Sep 15 12:15:08.260723 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.260682 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6"} err="failed to get container status \"637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6\": rpc error: code = NotFound desc = could not find container \"637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6\": container with ID starting with 637d5211d6b08eb35f0a39c56024258d1f3c05826ed8f19f1c4f193b699bbbe6 not found: ID does not exist" Sep 15 12:15:08.260723 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.260700 2579 scope.go:117] "RemoveContainer" containerID="06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f" Sep 15 12:15:08.268884 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.268867 2579 scope.go:117] "RemoveContainer" containerID="06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f" Sep 15 12:15:08.269167 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:15:08.269146 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f\": container with ID starting with 06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f not found: ID does not exist" containerID="06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f" Sep 15 12:15:08.269333 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.269179 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f"} err="failed to get container status \"06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f\": rpc error: code = NotFound desc = could not find container \"06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f\": container with ID starting with 06e44d1cee3058697df8c2dee9af337f4a609a75bb50afd7ac9be155913da64f not found: ID does not exist" Sep 15 12:15:08.269333 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.269215 2579 scope.go:117] "RemoveContainer" containerID="c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a" Sep 15 12:15:08.277029 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.277012 2579 scope.go:117] "RemoveContainer" containerID="c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a" Sep 15 12:15:08.277371 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:15:08.277350 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a\": container with ID starting with c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a not found: ID does not exist" containerID="c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a" Sep 15 12:15:08.277432 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.277379 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a"} err="failed to get container status \"c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a\": rpc error: code = NotFound desc = could not find container \"c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a\": container with ID starting with c1480ef301364441954a6a191b700ff855f0ea79caad5704ff7a8b8ee92fc09a not found: ID does not exist" Sep 15 12:15:08.285373 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfxgb\" (UniqueName: \"kubernetes.io/projected/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-kube-api-access-lfxgb\") pod \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\" (UID: \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\") " Sep 15 12:15:08.285482 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285395 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-metrics-certs\") pod \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " Sep 15 12:15:08.285482 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285415 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-openshift-service-ca\") pod \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " Sep 15 12:15:08.285482 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285468 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz4jd\" (UniqueName: \"kubernetes.io/projected/59840da1-db12-46e6-9289-2a2fb7de7e2b-kube-api-access-vz4jd\") pod \"59840da1-db12-46e6-9289-2a2fb7de7e2b\" (UID: \"59840da1-db12-46e6-9289-2a2fb7de7e2b\") " Sep 15 12:15:08.285705 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285488 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfq24\" (UniqueName: \"kubernetes.io/projected/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-kube-api-access-wfq24\") pod \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " Sep 15 12:15:08.285705 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285512 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls\") pod \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\" (UID: \"66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b\") " Sep 15 12:15:08.285705 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285548 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-plugins-config-volume\") pod \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\" (UID: \"76c6c366-72b2-40b8-ad2e-1d4b30a815b3\") " Sep 15 12:15:08.285705 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285571 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59840da1-db12-46e6-9289-2a2fb7de7e2b-plugins-config-volume\") pod \"59840da1-db12-46e6-9289-2a2fb7de7e2b\" (UID: \"59840da1-db12-46e6-9289-2a2fb7de7e2b\") " Sep 15 12:15:08.285912 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285874 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" (UID: "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:15:08.286002 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.285970 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76c6c366-72b2-40b8-ad2e-1d4b30a815b3" (UID: "76c6c366-72b2-40b8-ad2e-1d4b30a815b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:15:08.286300 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.286227 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59840da1-db12-46e6-9289-2a2fb7de7e2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59840da1-db12-46e6-9289-2a2fb7de7e2b" (UID: "59840da1-db12-46e6-9289-2a2fb7de7e2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:15:08.287738 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.287712 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" (UID: "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:15:08.287830 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.287742 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59840da1-db12-46e6-9289-2a2fb7de7e2b-kube-api-access-vz4jd" (OuterVolumeSpecName: "kube-api-access-vz4jd") pod "59840da1-db12-46e6-9289-2a2fb7de7e2b" (UID: "59840da1-db12-46e6-9289-2a2fb7de7e2b"). InnerVolumeSpecName "kube-api-access-vz4jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:15:08.287887 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.287870 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-kube-api-access-wfq24" (OuterVolumeSpecName: "kube-api-access-wfq24") pod "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" (UID: "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b"). InnerVolumeSpecName "kube-api-access-wfq24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:15:08.288233 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.288188 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" (UID: "66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:15:08.288233 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.288226 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-kube-api-access-lfxgb" (OuterVolumeSpecName: "kube-api-access-lfxgb") pod "76c6c366-72b2-40b8-ad2e-1d4b30a815b3" (UID: "76c6c366-72b2-40b8-ad2e-1d4b30a815b3"). InnerVolumeSpecName "kube-api-access-lfxgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:15:08.387005 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.386966 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxgb\" (UniqueName: \"kubernetes.io/projected/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-kube-api-access-lfxgb\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:15:08.387005 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.386998 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:15:08.387005 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.387007 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:15:08.387005 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.387016 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz4jd\" (UniqueName: \"kubernetes.io/projected/59840da1-db12-46e6-9289-2a2fb7de7e2b-kube-api-access-vz4jd\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:15:08.387322 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.387026 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfq24\" (UniqueName: \"kubernetes.io/projected/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-kube-api-access-wfq24\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:15:08.387322 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.387037 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:15:08.387322 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.387045 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76c6c366-72b2-40b8-ad2e-1d4b30a815b3-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:15:08.387322 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.387053 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59840da1-db12-46e6-9289-2a2fb7de7e2b-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:15:08.582581 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.582523 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt"] Sep 15 12:15:08.590799 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.586093 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7139fb-588c4885b6-p6vnt"] Sep 15 12:15:08.610877 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.610840 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh"] Sep 15 12:15:08.620514 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.620486 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7139fb-87479c56f-zdksh"] Sep 15 12:15:08.634128 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.634100 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6"] Sep 15 12:15:08.642788 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:08.642759 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7139fb-8d55ddfd6-f4mh6"] Sep 15 12:15:09.188648 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:09.188611 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59840da1-db12-46e6-9289-2a2fb7de7e2b" path="/var/lib/kubelet/pods/59840da1-db12-46e6-9289-2a2fb7de7e2b/volumes" Sep 15 12:15:09.189020 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:09.188988 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" path="/var/lib/kubelet/pods/66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b/volumes" Sep 15 12:15:09.189327 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:15:09.189313 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c6c366-72b2-40b8-ad2e-1d4b30a815b3" path="/var/lib/kubelet/pods/76c6c366-72b2-40b8-ad2e-1d4b30a815b3/volumes" Sep 15 12:16:11.055289 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.055191 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw"] Sep 15 12:16:11.055820 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.055510 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" podUID="57bae194-0323-440a-928e-3355bc1ce80f" containerName="maas-api" containerID="cri-o://0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d" gracePeriod=30 Sep 15 12:16:11.319481 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.319426 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:16:11.424548 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.424511 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg77n\" (UniqueName: \"kubernetes.io/projected/57bae194-0323-440a-928e-3355bc1ce80f-kube-api-access-mg77n\") pod \"57bae194-0323-440a-928e-3355bc1ce80f\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " Sep 15 12:16:11.424738 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.424596 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bae194-0323-440a-928e-3355bc1ce80f-openshift-service-ca\") pod \"57bae194-0323-440a-928e-3355bc1ce80f\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " Sep 15 12:16:11.424738 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.424619 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-metrics-certs\") pod \"57bae194-0323-440a-928e-3355bc1ce80f\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " Sep 15 12:16:11.424738 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.424669 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-maas-api-tls\") pod \"57bae194-0323-440a-928e-3355bc1ce80f\" (UID: \"57bae194-0323-440a-928e-3355bc1ce80f\") " Sep 15 12:16:11.424960 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.424939 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bae194-0323-440a-928e-3355bc1ce80f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57bae194-0323-440a-928e-3355bc1ce80f" (UID: "57bae194-0323-440a-928e-3355bc1ce80f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:16:11.426821 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.426787 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57bae194-0323-440a-928e-3355bc1ce80f" (UID: "57bae194-0323-440a-928e-3355bc1ce80f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:16:11.426924 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.426822 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57bae194-0323-440a-928e-3355bc1ce80f" (UID: "57bae194-0323-440a-928e-3355bc1ce80f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:16:11.426924 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.426885 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bae194-0323-440a-928e-3355bc1ce80f-kube-api-access-mg77n" (OuterVolumeSpecName: "kube-api-access-mg77n") pod "57bae194-0323-440a-928e-3355bc1ce80f" (UID: "57bae194-0323-440a-928e-3355bc1ce80f"). InnerVolumeSpecName "kube-api-access-mg77n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:16:11.525574 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.525539 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:11.525574 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.525566 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg77n\" (UniqueName: \"kubernetes.io/projected/57bae194-0323-440a-928e-3355bc1ce80f-kube-api-access-mg77n\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:11.525574 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.525575 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57bae194-0323-440a-928e-3355bc1ce80f-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:11.525793 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.525584 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57bae194-0323-440a-928e-3355bc1ce80f-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:11.565848 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.565818 2579 generic.go:356] "Generic (PLEG): container finished" podID="57bae194-0323-440a-928e-3355bc1ce80f" containerID="0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d" exitCode=0 Sep 15 12:16:11.566011 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.565861 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" Sep 15 12:16:11.566011 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.565902 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" event={"ID":"57bae194-0323-440a-928e-3355bc1ce80f","Type":"ContainerDied","Data":"0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d"} Sep 15 12:16:11.566011 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.565953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw" event={"ID":"57bae194-0323-440a-928e-3355bc1ce80f","Type":"ContainerDied","Data":"aee8a1eddff849cb474393a418648792a5e5f2115b50cde9d0038156bc35664f"} Sep 15 12:16:11.566011 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.566007 2579 scope.go:117] "RemoveContainer" containerID="0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d" Sep 15 12:16:11.575584 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.575570 2579 scope.go:117] "RemoveContainer" containerID="0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d" Sep 15 12:16:11.575872 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:16:11.575855 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d\": container with ID starting with 0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d not found: ID does not exist" containerID="0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d" Sep 15 12:16:11.575927 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.575881 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d"} err="failed to get container status \"0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d\": rpc error: code = NotFound desc = could not find container \"0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d\": container with ID starting with 0ff1032b38fe0f3c025ca89e0faee24a345149d425cbdebde7d3a0f288fa055d not found: ID does not exist" Sep 15 12:16:11.604788 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.604763 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw"] Sep 15 12:16:11.607278 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:11.607258 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74aca3-7f66bd4bdb-fcrcw"] Sep 15 12:16:13.187956 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:13.187921 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57bae194-0323-440a-928e-3355bc1ce80f" path="/var/lib/kubelet/pods/57bae194-0323-440a-928e-3355bc1ce80f/volumes" Sep 15 12:16:31.228074 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.228036 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7"] Sep 15 12:16:31.228684 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.228294 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" podUID="b457ae41-dab4-46f1-beb8-ab0896c241dc" containerName="payload-processing" containerID="cri-o://16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf" gracePeriod=30 Sep 15 12:16:31.233752 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.233677 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh"] Sep 15 12:16:31.233961 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.233934 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" podUID="bdadc4a5-cae2-4c64-bdda-b28b45c9a54b" containerName="payload-pre-processing" containerID="cri-o://5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b" gracePeriod=30 Sep 15 12:16:31.511347 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.511312 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:16:31.514454 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.514434 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:16:31.596463 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.596432 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwf74\" (UniqueName: \"kubernetes.io/projected/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-kube-api-access-dwf74\") pod \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\" (UID: \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\") " Sep 15 12:16:31.596637 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.596481 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457ae41-dab4-46f1-beb8-ab0896c241dc-plugins-config-volume\") pod \"b457ae41-dab4-46f1-beb8-ab0896c241dc\" (UID: \"b457ae41-dab4-46f1-beb8-ab0896c241dc\") " Sep 15 12:16:31.596637 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.596497 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-plugins-config-volume\") pod \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\" (UID: \"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b\") " Sep 15 12:16:31.596637 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.596533 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjxvs\" (UniqueName: \"kubernetes.io/projected/b457ae41-dab4-46f1-beb8-ab0896c241dc-kube-api-access-kjxvs\") pod \"b457ae41-dab4-46f1-beb8-ab0896c241dc\" (UID: \"b457ae41-dab4-46f1-beb8-ab0896c241dc\") " Sep 15 12:16:31.596861 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.596838 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b457ae41-dab4-46f1-beb8-ab0896c241dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b457ae41-dab4-46f1-beb8-ab0896c241dc" (UID: "b457ae41-dab4-46f1-beb8-ab0896c241dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:16:31.596942 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.596917 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bdadc4a5-cae2-4c64-bdda-b28b45c9a54b" (UID: "bdadc4a5-cae2-4c64-bdda-b28b45c9a54b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:16:31.598857 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.598828 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b457ae41-dab4-46f1-beb8-ab0896c241dc-kube-api-access-kjxvs" (OuterVolumeSpecName: "kube-api-access-kjxvs") pod "b457ae41-dab4-46f1-beb8-ab0896c241dc" (UID: "b457ae41-dab4-46f1-beb8-ab0896c241dc"). InnerVolumeSpecName "kube-api-access-kjxvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:16:31.599004 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.598979 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-kube-api-access-dwf74" (OuterVolumeSpecName: "kube-api-access-dwf74") pod "bdadc4a5-cae2-4c64-bdda-b28b45c9a54b" (UID: "bdadc4a5-cae2-4c64-bdda-b28b45c9a54b"). InnerVolumeSpecName "kube-api-access-dwf74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:16:31.670676 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.670641 2579 generic.go:356] "Generic (PLEG): container finished" podID="b457ae41-dab4-46f1-beb8-ab0896c241dc" containerID="16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf" exitCode=0 Sep 15 12:16:31.670854 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.670685 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" Sep 15 12:16:31.670854 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.670731 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" event={"ID":"b457ae41-dab4-46f1-beb8-ab0896c241dc","Type":"ContainerDied","Data":"16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf"} Sep 15 12:16:31.670854 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.670786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7" event={"ID":"b457ae41-dab4-46f1-beb8-ab0896c241dc","Type":"ContainerDied","Data":"c21692fad304f3bdd040aa35cfe3ac14f3c48d1ea4cc431965ef0d5316cfcc89"} Sep 15 12:16:31.670854 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.670816 2579 scope.go:117] "RemoveContainer" containerID="16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf" Sep 15 12:16:31.672308 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.672276 2579 generic.go:356] "Generic (PLEG): container finished" podID="bdadc4a5-cae2-4c64-bdda-b28b45c9a54b" containerID="5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b" exitCode=0 Sep 15 12:16:31.672417 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.672312 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" event={"ID":"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b","Type":"ContainerDied","Data":"5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b"} Sep 15 12:16:31.672417 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.672339 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" Sep 15 12:16:31.672417 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.672366 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh" event={"ID":"bdadc4a5-cae2-4c64-bdda-b28b45c9a54b","Type":"ContainerDied","Data":"54137f3a512022e7bdf5b6c8acced8ba50860b605ed279259cce4db7bd1fc4c8"} Sep 15 12:16:31.680246 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.680223 2579 scope.go:117] "RemoveContainer" containerID="16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf" Sep 15 12:16:31.680530 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:16:31.680512 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf\": container with ID starting with 16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf not found: ID does not exist" containerID="16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf" Sep 15 12:16:31.680614 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.680538 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf"} err="failed to get container status \"16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf\": rpc error: code = NotFound desc = could not find container \"16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf\": container with ID starting with 16f59b7642768b62dbb8429a6e14a7d68bdb709642fef9ce1c050689e631afdf not found: ID does not exist" Sep 15 12:16:31.680614 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.680556 2579 scope.go:117] "RemoveContainer" containerID="5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b" Sep 15 12:16:31.689146 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.689127 2579 scope.go:117] "RemoveContainer" containerID="5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b" Sep 15 12:16:31.689479 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:16:31.689460 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b\": container with ID starting with 5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b not found: ID does not exist" containerID="5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b" Sep 15 12:16:31.689530 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.689487 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b"} err="failed to get container status \"5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b\": rpc error: code = NotFound desc = could not find container \"5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b\": container with ID starting with 5559dc3e8f5cc978b5deab9af2f79d32b43c43dd7939cda60c6448f1c8cf664b not found: ID does not exist" Sep 15 12:16:31.697682 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.697656 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwf74\" (UniqueName: \"kubernetes.io/projected/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-kube-api-access-dwf74\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:31.697682 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.697683 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b457ae41-dab4-46f1-beb8-ab0896c241dc-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:31.697819 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.697697 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:31.697819 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.697707 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxvs\" (UniqueName: \"kubernetes.io/projected/b457ae41-dab4-46f1-beb8-ab0896c241dc-kube-api-access-kjxvs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:31.716194 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.716150 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh"] Sep 15 12:16:31.720330 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.720305 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2b3936-7894556c8b-st4nh"] Sep 15 12:16:31.739309 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.739231 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7"] Sep 15 12:16:31.749493 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:31.749465 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-2b3936-5dd97b5f7c-x5jr7"] Sep 15 12:16:33.196039 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:33.196008 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b457ae41-dab4-46f1-beb8-ab0896c241dc" path="/var/lib/kubelet/pods/b457ae41-dab4-46f1-beb8-ab0896c241dc/volumes" Sep 15 12:16:33.196411 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:33.196367 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdadc4a5-cae2-4c64-bdda-b28b45c9a54b" path="/var/lib/kubelet/pods/bdadc4a5-cae2-4c64-bdda-b28b45c9a54b/volumes" Sep 15 12:16:52.937316 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:52.937271 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x"] Sep 15 12:16:52.937927 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:52.937474 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" podUID="6c14dacd-ad80-485f-8dfa-a517dd6983e4" containerName="payload-processing" containerID="cri-o://4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff" gracePeriod=30 Sep 15 12:16:52.952097 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:52.952068 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n"] Sep 15 12:16:52.952698 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:52.952353 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" podUID="b5edb4e4-507f-4661-8969-13090d6a0fbf" containerName="payload-pre-processing" containerID="cri-o://83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad" gracePeriod=30 Sep 15 12:16:53.225456 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.225431 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:16:53.228152 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.228133 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:16:53.288410 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.288375 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5edb4e4-507f-4661-8969-13090d6a0fbf-plugins-config-volume\") pod \"b5edb4e4-507f-4661-8969-13090d6a0fbf\" (UID: \"b5edb4e4-507f-4661-8969-13090d6a0fbf\") " Sep 15 12:16:53.288578 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.288461 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbc77\" (UniqueName: \"kubernetes.io/projected/6c14dacd-ad80-485f-8dfa-a517dd6983e4-kube-api-access-gbc77\") pod \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\" (UID: \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\") " Sep 15 12:16:53.288578 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.288488 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c14dacd-ad80-485f-8dfa-a517dd6983e4-plugins-config-volume\") pod \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\" (UID: \"6c14dacd-ad80-485f-8dfa-a517dd6983e4\") " Sep 15 12:16:53.288578 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.288539 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bscn9\" (UniqueName: \"kubernetes.io/projected/b5edb4e4-507f-4661-8969-13090d6a0fbf-kube-api-access-bscn9\") pod \"b5edb4e4-507f-4661-8969-13090d6a0fbf\" (UID: \"b5edb4e4-507f-4661-8969-13090d6a0fbf\") " Sep 15 12:16:53.288846 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.288818 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5edb4e4-507f-4661-8969-13090d6a0fbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5edb4e4-507f-4661-8969-13090d6a0fbf" (UID: "b5edb4e4-507f-4661-8969-13090d6a0fbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:16:53.288910 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.288881 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c14dacd-ad80-485f-8dfa-a517dd6983e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c14dacd-ad80-485f-8dfa-a517dd6983e4" (UID: "6c14dacd-ad80-485f-8dfa-a517dd6983e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:16:53.290756 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.290735 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c14dacd-ad80-485f-8dfa-a517dd6983e4-kube-api-access-gbc77" (OuterVolumeSpecName: "kube-api-access-gbc77") pod "6c14dacd-ad80-485f-8dfa-a517dd6983e4" (UID: "6c14dacd-ad80-485f-8dfa-a517dd6983e4"). InnerVolumeSpecName "kube-api-access-gbc77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:16:53.290863 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.290820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5edb4e4-507f-4661-8969-13090d6a0fbf-kube-api-access-bscn9" (OuterVolumeSpecName: "kube-api-access-bscn9") pod "b5edb4e4-507f-4661-8969-13090d6a0fbf" (UID: "b5edb4e4-507f-4661-8969-13090d6a0fbf"). InnerVolumeSpecName "kube-api-access-bscn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:16:53.390140 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.390100 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5edb4e4-507f-4661-8969-13090d6a0fbf-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:53.390140 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.390133 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbc77\" (UniqueName: \"kubernetes.io/projected/6c14dacd-ad80-485f-8dfa-a517dd6983e4-kube-api-access-gbc77\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:53.390140 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.390142 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c14dacd-ad80-485f-8dfa-a517dd6983e4-plugins-config-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:53.390140 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.390151 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscn9\" (UniqueName: \"kubernetes.io/projected/b5edb4e4-507f-4661-8969-13090d6a0fbf-kube-api-access-bscn9\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:16:53.779258 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.779192 2579 generic.go:356] "Generic (PLEG): container finished" podID="b5edb4e4-507f-4661-8969-13090d6a0fbf" containerID="83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad" exitCode=0 Sep 15 12:16:53.779258 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.779246 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" event={"ID":"b5edb4e4-507f-4661-8969-13090d6a0fbf","Type":"ContainerDied","Data":"83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad"} Sep 15 12:16:53.779537 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.779275 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" Sep 15 12:16:53.779537 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.779302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n" event={"ID":"b5edb4e4-507f-4661-8969-13090d6a0fbf","Type":"ContainerDied","Data":"6de915434c8a6f17ea8cf1abdd0051f5df159abcf822c698f49b985431a11e0d"} Sep 15 12:16:53.779537 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.779333 2579 scope.go:117] "RemoveContainer" containerID="83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad" Sep 15 12:16:53.780665 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.780639 2579 generic.go:356] "Generic (PLEG): container finished" podID="6c14dacd-ad80-485f-8dfa-a517dd6983e4" containerID="4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff" exitCode=0 Sep 15 12:16:53.780788 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.780680 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" Sep 15 12:16:53.780788 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.780724 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" event={"ID":"6c14dacd-ad80-485f-8dfa-a517dd6983e4","Type":"ContainerDied","Data":"4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff"} Sep 15 12:16:53.780890 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.780787 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x" event={"ID":"6c14dacd-ad80-485f-8dfa-a517dd6983e4","Type":"ContainerDied","Data":"ce52b63f62b95d1325e1fe2c88f97e919551f8fc1551c85f000a7c4839e5102c"} Sep 15 12:16:53.790572 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.790509 2579 scope.go:117] "RemoveContainer" containerID="83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad" Sep 15 12:16:53.790869 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:16:53.790814 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad\": container with ID starting with 83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad not found: ID does not exist" containerID="83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad" Sep 15 12:16:53.790869 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.790847 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad"} err="failed to get container status \"83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad\": rpc error: code = NotFound desc = could not find container \"83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad\": container with ID starting with 83757ada1d72da13231d7a0af13a68889170e6df759456ae2ceb64e43b72b1ad not found: ID does not exist" Sep 15 12:16:53.791005 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.790870 2579 scope.go:117] "RemoveContainer" containerID="4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff" Sep 15 12:16:53.800468 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.800416 2579 scope.go:117] "RemoveContainer" containerID="4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff" Sep 15 12:16:53.800711 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:16:53.800693 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff\": container with ID starting with 4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff not found: ID does not exist" containerID="4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff" Sep 15 12:16:53.800765 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.800719 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff"} err="failed to get container status \"4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff\": rpc error: code = NotFound desc = could not find container \"4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff\": container with ID starting with 4e11f2d428767ce3fc50fb47329af86f1df829b6e7c193dc4580b8b62e658cff not found: ID does not exist" Sep 15 12:16:53.822604 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.822581 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x"] Sep 15 12:16:53.828609 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.828583 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-32bbbb-8494dd8855-kz22x"] Sep 15 12:16:53.846410 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.846388 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n"] Sep 15 12:16:53.859054 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:53.859030 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-32bbbb-6b9f96f76b-stx7n"] Sep 15 12:16:55.188108 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:55.188054 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c14dacd-ad80-485f-8dfa-a517dd6983e4" path="/var/lib/kubelet/pods/6c14dacd-ad80-485f-8dfa-a517dd6983e4/volumes" Sep 15 12:16:55.188503 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:16:55.188488 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5edb4e4-507f-4661-8969-13090d6a0fbf" path="/var/lib/kubelet/pods/b5edb4e4-507f-4661-8969-13090d6a0fbf/volumes" Sep 15 12:17:04.099474 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.099430 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777d95b559-qkjvn"] Sep 15 12:17:04.100040 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.099677 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-777d95b559-qkjvn" podUID="ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" containerName="manager" containerID="cri-o://a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f" gracePeriod=30 Sep 15 12:17:04.351853 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.351779 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 12:17:04.482482 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.482442 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnkn6\" (UniqueName: \"kubernetes.io/projected/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-kube-api-access-jnkn6\") pod \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " Sep 15 12:17:04.482696 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.482494 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-metrics-certs\") pod \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " Sep 15 12:17:04.482696 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.482551 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-webhook-cert\") pod \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\" (UID: \"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba\") " Sep 15 12:17:04.484865 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.484830 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" (UID: "ebf8de17-cd85-4b51-a039-eeed6ed8c3ba"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:17:04.484865 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.484851 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-kube-api-access-jnkn6" (OuterVolumeSpecName: "kube-api-access-jnkn6") pod "ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" (UID: "ebf8de17-cd85-4b51-a039-eeed6ed8c3ba"). InnerVolumeSpecName "kube-api-access-jnkn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:17:04.485025 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.484916 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" (UID: "ebf8de17-cd85-4b51-a039-eeed6ed8c3ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:17:04.583347 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.583308 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-webhook-cert\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:17:04.583347 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.583338 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnkn6\" (UniqueName: \"kubernetes.io/projected/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-kube-api-access-jnkn6\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:17:04.583347 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.583349 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:17:04.837724 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.837684 2579 generic.go:356] "Generic (PLEG): container finished" podID="ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" containerID="a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f" exitCode=0 Sep 15 12:17:04.837884 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.837730 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-qkjvn" Sep 15 12:17:04.837884 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.837770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-qkjvn" event={"ID":"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba","Type":"ContainerDied","Data":"a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f"} Sep 15 12:17:04.837884 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.837831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-qkjvn" event={"ID":"ebf8de17-cd85-4b51-a039-eeed6ed8c3ba","Type":"ContainerDied","Data":"f73dc71f46c9f525f373df719898d2cbf5315d7d3a6c931c64b3204bbcffb802"} Sep 15 12:17:04.837884 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.837860 2579 scope.go:117] "RemoveContainer" containerID="a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f" Sep 15 12:17:04.847652 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.847629 2579 scope.go:117] "RemoveContainer" containerID="a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f" Sep 15 12:17:04.847960 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:17:04.847940 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f\": container with ID starting with a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f not found: ID does not exist" containerID="a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f" Sep 15 12:17:04.848013 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.847974 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f"} err="failed to get container status \"a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f\": rpc error: code = NotFound desc = could not find container \"a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f\": container with ID starting with a933bda7a8c676814b6f0a624669e0925c5974aab2f9e88e6833309b3dd7287f not found: ID does not exist" Sep 15 12:17:04.862748 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.862717 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777d95b559-qkjvn"] Sep 15 12:17:04.871088 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:04.871063 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-777d95b559-qkjvn"] Sep 15 12:17:05.187912 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:17:05.187882 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" path="/var/lib/kubelet/pods/ebf8de17-cd85-4b51-a039-eeed6ed8c3ba/volumes" Sep 15 12:18:05.683969 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.683934 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-777d95b559-qzxwl"] Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684304 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87e88d09-f545-474f-a4b0-77d7a651aa5f" containerName="payload-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684317 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="87e88d09-f545-474f-a4b0-77d7a651aa5f" containerName="payload-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684326 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5edb4e4-507f-4661-8969-13090d6a0fbf" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684332 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5edb4e4-507f-4661-8969-13090d6a0fbf" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684341 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdadc4a5-cae2-4c64-bdda-b28b45c9a54b" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684346 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdadc4a5-cae2-4c64-bdda-b28b45c9a54b" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684355 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59840da1-db12-46e6-9289-2a2fb7de7e2b" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684360 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="59840da1-db12-46e6-9289-2a2fb7de7e2b" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684370 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b457ae41-dab4-46f1-beb8-ab0896c241dc" containerName="payload-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684375 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b457ae41-dab4-46f1-beb8-ab0896c241dc" containerName="payload-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684384 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bae194-0323-440a-928e-3355bc1ce80f" containerName="maas-api" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684389 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bae194-0323-440a-928e-3355bc1ce80f" containerName="maas-api" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684396 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" containerName="maas-api" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684401 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" containerName="maas-api" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684406 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c14dacd-ad80-485f-8dfa-a517dd6983e4" containerName="payload-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684411 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c14dacd-ad80-485f-8dfa-a517dd6983e4" containerName="payload-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684419 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" containerName="manager" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684424 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" containerName="manager" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684429 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c6c366-72b2-40b8-ad2e-1d4b30a815b3" containerName="payload-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684434 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c6c366-72b2-40b8-ad2e-1d4b30a815b3" containerName="payload-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684441 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2877f62e-7c68-4c76-a728-d4005a4c6bab" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684448 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2877f62e-7c68-4c76-a728-d4005a4c6bab" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684453 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e2a905-94ce-4df8-bc53-8bb681e64b18" containerName="maas-api" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684459 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e2a905-94ce-4df8-bc53-8bb681e64b18" containerName="maas-api" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684505 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e2a905-94ce-4df8-bc53-8bb681e64b18" containerName="maas-api" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684514 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5edb4e4-507f-4661-8969-13090d6a0fbf" containerName="payload-pre-processing" Sep 15 12:18:05.684506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684520 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c14dacd-ad80-485f-8dfa-a517dd6983e4" containerName="payload-processing" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684527 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdadc4a5-cae2-4c64-bdda-b28b45c9a54b" containerName="payload-pre-processing" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684532 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c6c366-72b2-40b8-ad2e-1d4b30a815b3" containerName="payload-processing" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684536 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bae194-0323-440a-928e-3355bc1ce80f" containerName="maas-api" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684542 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="66fb8ac4-7b3a-4fe8-b34d-e5110bb34f6b" containerName="maas-api" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684546 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="87e88d09-f545-474f-a4b0-77d7a651aa5f" containerName="payload-processing" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684551 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="59840da1-db12-46e6-9289-2a2fb7de7e2b" containerName="payload-pre-processing" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684556 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2877f62e-7c68-4c76-a728-d4005a4c6bab" containerName="payload-pre-processing" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684562 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf8de17-cd85-4b51-a039-eeed6ed8c3ba" containerName="manager" Sep 15 12:18:05.685401 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.684566 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b457ae41-dab4-46f1-beb8-ab0896c241dc" containerName="payload-processing" Sep 15 12:18:05.687475 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.687454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.691417 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.691394 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 15 12:18:05.691541 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.691394 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 15 12:18:05.692657 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.692640 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rnnhv\"" Sep 15 12:18:05.692769 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.692685 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 15 12:18:05.700596 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.700575 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777d95b559-qzxwl"] Sep 15 12:18:05.710292 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.710258 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87qff\" (UniqueName: \"kubernetes.io/projected/d77862d9-37ab-4bfb-bf89-431b23e902c7-kube-api-access-87qff\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.710416 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.710331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-metrics-certs\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.710416 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.710388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-webhook-cert\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.811150 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.811106 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-webhook-cert\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.811391 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.811185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87qff\" (UniqueName: \"kubernetes.io/projected/d77862d9-37ab-4bfb-bf89-431b23e902c7-kube-api-access-87qff\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.811391 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.811285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-metrics-certs\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.813745 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.813720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-metrics-certs\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.813867 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.813846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-webhook-cert\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.821558 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.821534 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87qff\" (UniqueName: \"kubernetes.io/projected/d77862d9-37ab-4bfb-bf89-431b23e902c7-kube-api-access-87qff\") pod \"maas-controller-777d95b559-qzxwl\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:05.998303 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:05.998193 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:06.272626 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:06.272602 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777d95b559-qzxwl"] Sep 15 12:18:06.274839 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:18:06.274808 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77862d9_37ab_4bfb_bf89_431b23e902c7.slice/crio-cc1e773cdc86903385a502ecf399d72159445b8f439ad2b8a19549a621c6b6d9 WatchSource:0}: Error finding container cc1e773cdc86903385a502ecf399d72159445b8f439ad2b8a19549a621c6b6d9: Status 404 returned error can't find the container with id cc1e773cdc86903385a502ecf399d72159445b8f439ad2b8a19549a621c6b6d9 Sep 15 12:18:06.276130 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:06.276114 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:18:07.141837 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:07.141736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-qzxwl" event={"ID":"d77862d9-37ab-4bfb-bf89-431b23e902c7","Type":"ContainerStarted","Data":"f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5"} Sep 15 12:18:07.141837 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:07.141798 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-qzxwl" event={"ID":"d77862d9-37ab-4bfb-bf89-431b23e902c7","Type":"ContainerStarted","Data":"cc1e773cdc86903385a502ecf399d72159445b8f439ad2b8a19549a621c6b6d9"} Sep 15 12:18:07.141837 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:07.141823 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:07.163777 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:07.163721 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-777d95b559-qzxwl" podStartSLOduration=1.727118955 podStartE2EDuration="2.163704119s" totalImagesPullingTime="436.585164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:18:07.162853578 +0000 UTC m=+2374.507962185" watchObservedRunningTime="2026-09-15 12:18:07.163704119 +0000 UTC m=+2374.508812725" Sep 15 12:18:18.149804 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:18.149769 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:27.510527 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.510488 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777d95b559-qzxwl"] Sep 15 12:18:27.511055 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.510750 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-777d95b559-qzxwl" podUID="d77862d9-37ab-4bfb-bf89-431b23e902c7" containerName="manager" containerID="cri-o://f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5" gracePeriod=30 Sep 15 12:18:27.757719 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.757695 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:27.784847 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.784776 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-webhook-cert\") pod \"d77862d9-37ab-4bfb-bf89-431b23e902c7\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " Sep 15 12:18:27.784847 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.784830 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87qff\" (UniqueName: \"kubernetes.io/projected/d77862d9-37ab-4bfb-bf89-431b23e902c7-kube-api-access-87qff\") pod \"d77862d9-37ab-4bfb-bf89-431b23e902c7\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " Sep 15 12:18:27.785051 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.784913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-metrics-certs\") pod \"d77862d9-37ab-4bfb-bf89-431b23e902c7\" (UID: \"d77862d9-37ab-4bfb-bf89-431b23e902c7\") " Sep 15 12:18:27.787053 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.787020 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d77862d9-37ab-4bfb-bf89-431b23e902c7" (UID: "d77862d9-37ab-4bfb-bf89-431b23e902c7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:18:27.787174 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.787112 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77862d9-37ab-4bfb-bf89-431b23e902c7-kube-api-access-87qff" (OuterVolumeSpecName: "kube-api-access-87qff") pod "d77862d9-37ab-4bfb-bf89-431b23e902c7" (UID: "d77862d9-37ab-4bfb-bf89-431b23e902c7"). InnerVolumeSpecName "kube-api-access-87qff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:18:27.787174 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.787154 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d77862d9-37ab-4bfb-bf89-431b23e902c7" (UID: "d77862d9-37ab-4bfb-bf89-431b23e902c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:18:27.886492 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.886453 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:18:27.886492 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.886485 2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d77862d9-37ab-4bfb-bf89-431b23e902c7-webhook-cert\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:18:27.886492 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:27.886498 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87qff\" (UniqueName: \"kubernetes.io/projected/d77862d9-37ab-4bfb-bf89-431b23e902c7-kube-api-access-87qff\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:18:28.243267 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.243236 2579 generic.go:356] "Generic (PLEG): container finished" podID="d77862d9-37ab-4bfb-bf89-431b23e902c7" containerID="f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5" exitCode=0 Sep 15 12:18:28.243450 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.243289 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-qzxwl" Sep 15 12:18:28.243450 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.243331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-qzxwl" event={"ID":"d77862d9-37ab-4bfb-bf89-431b23e902c7","Type":"ContainerDied","Data":"f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5"} Sep 15 12:18:28.243450 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.243382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-qzxwl" event={"ID":"d77862d9-37ab-4bfb-bf89-431b23e902c7","Type":"ContainerDied","Data":"cc1e773cdc86903385a502ecf399d72159445b8f439ad2b8a19549a621c6b6d9"} Sep 15 12:18:28.243450 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.243404 2579 scope.go:117] "RemoveContainer" containerID="f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5" Sep 15 12:18:28.252764 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.252745 2579 scope.go:117] "RemoveContainer" containerID="f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5" Sep 15 12:18:28.253064 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:18:28.253041 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5\": container with ID starting with f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5 not found: ID does not exist" containerID="f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5" Sep 15 12:18:28.253145 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.253073 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5"} err="failed to get container status \"f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5\": rpc error: code = NotFound desc = could not find container \"f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5\": container with ID starting with f886843fd7ba5e0f9c7bbc1e6366195c35025666a0d81e9088d0f5ec43aacce5 not found: ID does not exist" Sep 15 12:18:28.266658 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.266632 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-777d95b559-qzxwl"] Sep 15 12:18:28.271915 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:28.271894 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-777d95b559-qzxwl"] Sep 15 12:18:29.097592 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.097535 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-777d95b559-z5d55"] Sep 15 12:18:29.098021 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.097997 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77862d9-37ab-4bfb-bf89-431b23e902c7" containerName="manager" Sep 15 12:18:29.098021 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.098013 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77862d9-37ab-4bfb-bf89-431b23e902c7" containerName="manager" Sep 15 12:18:29.098155 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.098113 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77862d9-37ab-4bfb-bf89-431b23e902c7" containerName="manager" Sep 15 12:18:29.102289 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.102268 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.105974 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.105950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rnnhv\"" Sep 15 12:18:29.106108 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.105950 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 15 12:18:29.106108 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.105951 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 15 12:18:29.106108 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.106017 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 15 12:18:29.118825 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.118799 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777d95b559-z5d55"] Sep 15 12:18:29.187573 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.187537 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d77862d9-37ab-4bfb-bf89-431b23e902c7" path="/var/lib/kubelet/pods/d77862d9-37ab-4bfb-bf89-431b23e902c7/volumes" Sep 15 12:18:29.198884 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.198858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35290200-3990-4083-9f38-5452dc8b075a-metrics-certs\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.198986 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.198888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvk2z\" (UniqueName: \"kubernetes.io/projected/35290200-3990-4083-9f38-5452dc8b075a-kube-api-access-pvk2z\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.198986 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.198917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35290200-3990-4083-9f38-5452dc8b075a-webhook-cert\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.300291 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.300253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35290200-3990-4083-9f38-5452dc8b075a-metrics-certs\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.300291 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.300292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvk2z\" (UniqueName: \"kubernetes.io/projected/35290200-3990-4083-9f38-5452dc8b075a-kube-api-access-pvk2z\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.300552 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.300316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35290200-3990-4083-9f38-5452dc8b075a-webhook-cert\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.303072 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.303045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35290200-3990-4083-9f38-5452dc8b075a-webhook-cert\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.303186 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.303045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35290200-3990-4083-9f38-5452dc8b075a-metrics-certs\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.311333 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.311307 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvk2z\" (UniqueName: \"kubernetes.io/projected/35290200-3990-4083-9f38-5452dc8b075a-kube-api-access-pvk2z\") pod \"maas-controller-777d95b559-z5d55\" (UID: \"35290200-3990-4083-9f38-5452dc8b075a\") " pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.412861 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.412781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:29.691365 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:29.691340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-777d95b559-z5d55"] Sep 15 12:18:29.693098 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:18:29.693073 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35290200_3990_4083_9f38_5452dc8b075a.slice/crio-07e92ca3c8ddaa843e5305f395aace59fd74c51abc64fe3eeea20f300a521738 WatchSource:0}: Error finding container 07e92ca3c8ddaa843e5305f395aace59fd74c51abc64fe3eeea20f300a521738: Status 404 returned error can't find the container with id 07e92ca3c8ddaa843e5305f395aace59fd74c51abc64fe3eeea20f300a521738 Sep 15 12:18:30.256454 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:30.256411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-z5d55" event={"ID":"35290200-3990-4083-9f38-5452dc8b075a","Type":"ContainerStarted","Data":"0bdead1f7d327434e722ef25ec1af5d6e0207861fa1a8e2d27dcda5b75dda58f"} Sep 15 12:18:30.256909 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:30.256480 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:18:30.256909 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:30.256497 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-777d95b559-z5d55" event={"ID":"35290200-3990-4083-9f38-5452dc8b075a","Type":"ContainerStarted","Data":"07e92ca3c8ddaa843e5305f395aace59fd74c51abc64fe3eeea20f300a521738"} Sep 15 12:18:30.278356 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:30.278307 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-777d95b559-z5d55" podStartSLOduration=0.921159028 podStartE2EDuration="1.278293669s" totalImagesPullingTime="357.134641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:18:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:18:30.277378071 +0000 UTC m=+2397.622486690" watchObservedRunningTime="2026-09-15 12:18:30.278293669 +0000 UTC m=+2397.623402278" Sep 15 12:18:33.314295 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:33.314264 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:18:33.321878 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:33.321848 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:18:41.264164 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:18:41.264128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-777d95b559-z5d55" Sep 15 12:19:37.329911 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:37.329873 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8"] Sep 15 12:19:37.330421 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:37.330125 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" podUID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerName="manager" containerID="cri-o://6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2" gracePeriod=10 Sep 15 12:19:38.505216 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.505180 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 12:19:38.579887 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.579792 2579 generic.go:356] "Generic (PLEG): container finished" podID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerID="6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2" exitCode=0 Sep 15 12:19:38.579887 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.579846 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" Sep 15 12:19:38.579887 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.579873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" event={"ID":"1b8589f0-8c9f-47eb-ba14-4e9578f13774","Type":"ContainerDied","Data":"6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2"} Sep 15 12:19:38.580125 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.579934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8" event={"ID":"1b8589f0-8c9f-47eb-ba14-4e9578f13774","Type":"ContainerDied","Data":"40eee8a2dd7b653c75511ed30e0d6c840181afbf5e28d5719548ee8f41095001"} Sep 15 12:19:38.580125 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.579964 2579 scope.go:117] "RemoveContainer" containerID="6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2" Sep 15 12:19:38.589213 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.589181 2579 scope.go:117] "RemoveContainer" containerID="c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0" Sep 15 12:19:38.594477 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.594449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwwsq\" (UniqueName: \"kubernetes.io/projected/1b8589f0-8c9f-47eb-ba14-4e9578f13774-kube-api-access-kwwsq\") pod \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\" (UID: \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\") " Sep 15 12:19:38.594596 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.594565 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b8589f0-8c9f-47eb-ba14-4e9578f13774-extensions-socket-volume\") pod \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\" (UID: \"1b8589f0-8c9f-47eb-ba14-4e9578f13774\") " Sep 15 12:19:38.594923 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.594898 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b8589f0-8c9f-47eb-ba14-4e9578f13774-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1b8589f0-8c9f-47eb-ba14-4e9578f13774" (UID: "1b8589f0-8c9f-47eb-ba14-4e9578f13774"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 15 12:19:38.596626 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.596600 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8589f0-8c9f-47eb-ba14-4e9578f13774-kube-api-access-kwwsq" (OuterVolumeSpecName: "kube-api-access-kwwsq") pod "1b8589f0-8c9f-47eb-ba14-4e9578f13774" (UID: "1b8589f0-8c9f-47eb-ba14-4e9578f13774"). InnerVolumeSpecName "kube-api-access-kwwsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:19:38.597836 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.597809 2579 scope.go:117] "RemoveContainer" containerID="6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2" Sep 15 12:19:38.598194 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:19:38.598165 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2\": container with ID starting with 6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2 not found: ID does not exist" containerID="6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2" Sep 15 12:19:38.598299 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.598228 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2"} err="failed to get container status \"6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2\": rpc error: code = NotFound desc = could not find container \"6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2\": container with ID starting with 6dcf3cf3f06e5d70534dd418b13c83d0d5f939bfbabc218fc93c540bd0c77fe2 not found: ID does not exist" Sep 15 12:19:38.598299 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.598256 2579 scope.go:117] "RemoveContainer" containerID="c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0" Sep 15 12:19:38.598575 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:19:38.598558 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0\": container with ID starting with c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0 not found: ID does not exist" containerID="c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0" Sep 15 12:19:38.598629 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.598588 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0"} err="failed to get container status \"c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0\": rpc error: code = NotFound desc = could not find container \"c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0\": container with ID starting with c9381aa0e6902eed8a97336cf2b9a915449676c57c89e1f48d5db9a189ebbfb0 not found: ID does not exist" Sep 15 12:19:38.695221 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.695174 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b8589f0-8c9f-47eb-ba14-4e9578f13774-extensions-socket-volume\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:19:38.695403 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.695237 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwwsq\" (UniqueName: \"kubernetes.io/projected/1b8589f0-8c9f-47eb-ba14-4e9578f13774-kube-api-access-kwwsq\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:19:38.904925 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.904892 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8"] Sep 15 12:19:38.910286 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:38.910253 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-b6nb8"] Sep 15 12:19:39.188506 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:19:39.188474 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" path="/var/lib/kubelet/pods/1b8589f0-8c9f-47eb-ba14-4e9578f13774/volumes" Sep 15 12:20:38.021727 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.021688 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4"] Sep 15 12:20:38.022153 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.022011 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerName="manager" Sep 15 12:20:38.022153 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.022022 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerName="manager" Sep 15 12:20:38.022153 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.022036 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerName="manager" Sep 15 12:20:38.022153 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.022041 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerName="manager" Sep 15 12:20:38.022153 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.022107 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerName="manager" Sep 15 12:20:38.025098 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.025081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.027673 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.027654 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xwdp5\"" Sep 15 12:20:38.040060 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.040033 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4"] Sep 15 12:20:38.086487 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.086449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac4e1fb8-80e5-42f6-8314-4f1adb03e97a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-48pn4\" (UID: \"ac4e1fb8-80e5-42f6-8314-4f1adb03e97a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.086662 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.086569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpcxm\" (UniqueName: \"kubernetes.io/projected/ac4e1fb8-80e5-42f6-8314-4f1adb03e97a-kube-api-access-xpcxm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-48pn4\" (UID: \"ac4e1fb8-80e5-42f6-8314-4f1adb03e97a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.187691 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.187655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpcxm\" (UniqueName: \"kubernetes.io/projected/ac4e1fb8-80e5-42f6-8314-4f1adb03e97a-kube-api-access-xpcxm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-48pn4\" (UID: \"ac4e1fb8-80e5-42f6-8314-4f1adb03e97a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.187853 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.187704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac4e1fb8-80e5-42f6-8314-4f1adb03e97a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-48pn4\" (UID: \"ac4e1fb8-80e5-42f6-8314-4f1adb03e97a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.188041 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.188027 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac4e1fb8-80e5-42f6-8314-4f1adb03e97a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-48pn4\" (UID: \"ac4e1fb8-80e5-42f6-8314-4f1adb03e97a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.199724 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.199692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpcxm\" (UniqueName: \"kubernetes.io/projected/ac4e1fb8-80e5-42f6-8314-4f1adb03e97a-kube-api-access-xpcxm\") pod \"kuadrant-operator-controller-manager-5769f99d9f-48pn4\" (UID: \"ac4e1fb8-80e5-42f6-8314-4f1adb03e97a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.334728 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.334647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.616660 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.616594 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4"] Sep 15 12:20:38.619280 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:20:38.619251 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4e1fb8_80e5_42f6_8314_4f1adb03e97a.slice/crio-743bbbf032f32ac80a8658b70a0f20a24b5dd93ee537fb505a93576d15cccb1d WatchSource:0}: Error finding container 743bbbf032f32ac80a8658b70a0f20a24b5dd93ee537fb505a93576d15cccb1d: Status 404 returned error can't find the container with id 743bbbf032f32ac80a8658b70a0f20a24b5dd93ee537fb505a93576d15cccb1d Sep 15 12:20:38.868408 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.868324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" event={"ID":"ac4e1fb8-80e5-42f6-8314-4f1adb03e97a","Type":"ContainerStarted","Data":"24d95187b7d45056deb41b403f86c4b655b6e27b229e89b025c006856f4972cd"} Sep 15 12:20:38.868408 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.868374 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" event={"ID":"ac4e1fb8-80e5-42f6-8314-4f1adb03e97a","Type":"ContainerStarted","Data":"743bbbf032f32ac80a8658b70a0f20a24b5dd93ee537fb505a93576d15cccb1d"} Sep 15 12:20:38.868590 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.868456 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:20:38.894997 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:38.894937 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" podStartSLOduration=1.894915298 podStartE2EDuration="1.894915298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:20:38.893614636 +0000 UTC m=+2526.238723240" watchObservedRunningTime="2026-09-15 12:20:38.894915298 +0000 UTC m=+2526.240023905" Sep 15 12:20:49.876017 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:20:49.875984 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-48pn4" Sep 15 12:22:36.945194 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:36.945154 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84696d6768-c6m6h"] Sep 15 12:22:36.945675 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:36.945617 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8589f0-8c9f-47eb-ba14-4e9578f13774" containerName="manager" Sep 15 12:22:36.948582 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:36.948557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:36.960804 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:36.960770 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84696d6768-c6m6h"] Sep 15 12:22:37.043899 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.043865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mxl\" (UniqueName: \"kubernetes.io/projected/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-kube-api-access-49mxl\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.044103 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.043932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-metrics-certs\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.044103 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.044004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-maas-api-tls\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.044103 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.044066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-openshift-service-ca\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.144535 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.144488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-maas-api-tls\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.144535 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.144540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-openshift-service-ca\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.144820 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.144575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mxl\" (UniqueName: \"kubernetes.io/projected/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-kube-api-access-49mxl\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.144820 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.144646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-metrics-certs\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.145522 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.145490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-openshift-service-ca\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.147443 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.147421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-metrics-certs\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.147566 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.147421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-maas-api-tls\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.157809 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.157781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mxl\" (UniqueName: \"kubernetes.io/projected/eaec7673-59a8-4f8e-9ea2-08c0d4e18f52-kube-api-access-49mxl\") pod \"maas-api-84696d6768-c6m6h\" (UID: \"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52\") " pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.259277 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.259154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:37.544604 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:37.544388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84696d6768-c6m6h"] Sep 15 12:22:37.547503 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:22:37.547474 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaec7673_59a8_4f8e_9ea2_08c0d4e18f52.slice/crio-bea5a2d0fa1c15109060afe5879df5c7119f2a31503bafa474e98f40cba25ec5 WatchSource:0}: Error finding container bea5a2d0fa1c15109060afe5879df5c7119f2a31503bafa474e98f40cba25ec5: Status 404 returned error can't find the container with id bea5a2d0fa1c15109060afe5879df5c7119f2a31503bafa474e98f40cba25ec5 Sep 15 12:22:38.435952 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:38.435900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" event={"ID":"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52","Type":"ContainerStarted","Data":"af7a70ad38582b49f791fdd62f6b05f4a0720eeefad2134a78092e5e891dfd56"} Sep 15 12:22:38.436436 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:38.435972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" event={"ID":"eaec7673-59a8-4f8e-9ea2-08c0d4e18f52","Type":"ContainerStarted","Data":"bea5a2d0fa1c15109060afe5879df5c7119f2a31503bafa474e98f40cba25ec5"} Sep 15 12:22:38.436436 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:38.436019 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:38.456822 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:38.456767 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" podStartSLOduration=2.049568448 podStartE2EDuration="2.456747859s" totalImagesPullingTime="407.179411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:22:38.456442602 +0000 UTC m=+2645.801551209" watchObservedRunningTime="2026-09-15 12:22:38.456747859 +0000 UTC m=+2645.801856464" Sep 15 12:22:44.447112 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:44.447079 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84696d6768-c6m6h" Sep 15 12:22:52.094284 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:52.094242 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg"] Sep 15 12:22:52.094781 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:52.094478 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" podUID="ebd13dcd-d3a3-42eb-807b-6d43727411d4" containerName="maas-api" containerID="cri-o://ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f" gracePeriod=30 Sep 15 12:22:53.353753 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.353714 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 12:22:53.388742 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.388709 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-maas-api-tls\") pod \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " Sep 15 12:22:53.388932 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.388776 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl2lp\" (UniqueName: \"kubernetes.io/projected/ebd13dcd-d3a3-42eb-807b-6d43727411d4-kube-api-access-sl2lp\") pod \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " Sep 15 12:22:53.388932 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.388804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-metrics-certs\") pod \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " Sep 15 12:22:53.388932 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.388863 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd13dcd-d3a3-42eb-807b-6d43727411d4-openshift-service-ca\") pod \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\" (UID: \"ebd13dcd-d3a3-42eb-807b-6d43727411d4\") " Sep 15 12:22:53.389612 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.389506 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd13dcd-d3a3-42eb-807b-6d43727411d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebd13dcd-d3a3-42eb-807b-6d43727411d4" (UID: "ebd13dcd-d3a3-42eb-807b-6d43727411d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 15 12:22:53.391254 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.391194 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd13dcd-d3a3-42eb-807b-6d43727411d4-kube-api-access-sl2lp" (OuterVolumeSpecName: "kube-api-access-sl2lp") pod "ebd13dcd-d3a3-42eb-807b-6d43727411d4" (UID: "ebd13dcd-d3a3-42eb-807b-6d43727411d4"). InnerVolumeSpecName "kube-api-access-sl2lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 15 12:22:53.391254 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.391241 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebd13dcd-d3a3-42eb-807b-6d43727411d4" (UID: "ebd13dcd-d3a3-42eb-807b-6d43727411d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:22:53.391254 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.391251 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ebd13dcd-d3a3-42eb-807b-6d43727411d4" (UID: "ebd13dcd-d3a3-42eb-807b-6d43727411d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 15 12:22:53.490376 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.490282 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl2lp\" (UniqueName: \"kubernetes.io/projected/ebd13dcd-d3a3-42eb-807b-6d43727411d4-kube-api-access-sl2lp\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:22:53.490376 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.490319 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-metrics-certs\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:22:53.490376 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.490328 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd13dcd-d3a3-42eb-807b-6d43727411d4-openshift-service-ca\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:22:53.490376 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.490337 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd13dcd-d3a3-42eb-807b-6d43727411d4-maas-api-tls\") on node \"ip-10-0-140-16.ec2.internal\" DevicePath \"\"" Sep 15 12:22:53.512528 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.512490 2579 generic.go:356] "Generic (PLEG): container finished" podID="ebd13dcd-d3a3-42eb-807b-6d43727411d4" containerID="ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f" exitCode=0 Sep 15 12:22:53.512709 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.512536 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" Sep 15 12:22:53.512709 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.512578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" event={"ID":"ebd13dcd-d3a3-42eb-807b-6d43727411d4","Type":"ContainerDied","Data":"ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f"} Sep 15 12:22:53.512709 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.512633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg" event={"ID":"ebd13dcd-d3a3-42eb-807b-6d43727411d4","Type":"ContainerDied","Data":"2b4ad0966e05c7a2a61e8697793b70bf23fd5a556ed324154f9d17020f806230"} Sep 15 12:22:53.512709 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.512656 2579 scope.go:117] "RemoveContainer" containerID="ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f" Sep 15 12:22:53.522300 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.522282 2579 scope.go:117] "RemoveContainer" containerID="ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f" Sep 15 12:22:53.522642 ip-10-0-140-16 kubenswrapper[2579]: E0915 12:22:53.522620 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f\": container with ID starting with ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f not found: ID does not exist" containerID="ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f" Sep 15 12:22:53.522726 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.522653 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f"} err="failed to get container status \"ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f\": rpc error: code = NotFound desc = could not find container \"ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f\": container with ID starting with ccc2e3ce86b1d22da000be6114440951ebf237addb3245879d50fe968a68426f not found: ID does not exist" Sep 15 12:22:53.536492 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.536442 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg"] Sep 15 12:22:53.541339 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:53.541291 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c49d8454d-xsmjg"] Sep 15 12:22:55.188025 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:22:55.187978 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd13dcd-d3a3-42eb-807b-6d43727411d4" path="/var/lib/kubelet/pods/ebd13dcd-d3a3-42eb-807b-6d43727411d4/volumes" Sep 15 12:23:10.023925 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:10.023887 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cb98cc9fc-hsx8x_dc3fbecd-5eeb-4855-9b26-b218c74137ea/authorino/0.log" Sep 15 12:23:15.486447 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:15.486416 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lknpt_621c0998-47bc-4da1-8be5-b20fdebdef73/manager/0.log" Sep 15 12:23:15.606780 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:15.606748 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-n2zwf_83dbb156-d8db-496a-8b6d-273d8b71a7d6/manager/0.log" Sep 15 12:23:15.728821 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:15.728787 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-777d95b559-z5d55_35290200-3990-4083-9f38-5452dc8b075a/manager/0.log" Sep 15 12:23:16.247103 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:16.247070 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66bc7bc78c-lpkmq_5204c2ed-93f0-40c7-a024-855959b4f0a0/copy-manifests/0.log" Sep 15 12:23:16.256280 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:16.256236 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66bc7bc78c-lpkmq_5204c2ed-93f0-40c7-a024-855959b4f0a0/manager/0.log" Sep 15 12:23:16.836666 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:16.836634 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84696d6768-c6m6h_eaec7673-59a8-4f8e-9ea2-08c0d4e18f52/maas-api/0.log" Sep 15 12:23:17.176805 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:17.176714 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hbbjr_febc6a96-0392-44b4-ab90-32c16d2f5a82/postgres/0.log" Sep 15 12:23:17.953991 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:17.953958 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cb98cc9fc-hsx8x_dc3fbecd-5eeb-4855-9b26-b218c74137ea/authorino/0.log" Sep 15 12:23:18.074393 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:18.074360 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-znvd5_93df8817-6a9f-4663-bb6b-2fc8d6981b97/manager/0.log" Sep 15 12:23:18.192604 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:18.192574 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-fgnnv_d069f28d-a8d6-4393-af3c-748c149d934f/manager/0.log" Sep 15 12:23:18.306275 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:18.306169 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n28h4_35bc74e8-130c-45c7-aec2-c0b72f02fb90/kuadrant-console-plugin/0.log" Sep 15 12:23:18.430995 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:18.430961 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-48pn4_ac4e1fb8-80e5-42f6-8314-4f1adb03e97a/manager/0.log" Sep 15 12:23:18.678876 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:18.678845 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-clmqc_39e7a5ff-de53-4728-81a0-d9f795c4fa47/manager/0.log" Sep 15 12:23:19.290061 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:19.290028 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-df8dfc748-4sgj9_efdd1b29-a854-4729-9b69-b1819af7d102/kube-auth-proxy/0.log" Sep 15 12:23:19.640322 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:19.640224 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-j2rxj_c6f5fe03-607a-4f4a-9a59-f176bae5c3cc/payload-pre-processing/0.log" Sep 15 12:23:20.229146 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:20.229114 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6fhs6_7300a517-6504-4680-a82e-f847ae0283a4/main/0.log" Sep 15 12:23:26.813741 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.813649 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q4mgh/must-gather-h76n6"] Sep 15 12:23:26.814320 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.814180 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd13dcd-d3a3-42eb-807b-6d43727411d4" containerName="maas-api" Sep 15 12:23:26.814320 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.814219 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd13dcd-d3a3-42eb-807b-6d43727411d4" containerName="maas-api" Sep 15 12:23:26.814320 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.814302 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd13dcd-d3a3-42eb-807b-6d43727411d4" containerName="maas-api" Sep 15 12:23:26.818296 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.818266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q4mgh/must-gather-h76n6" Sep 15 12:23:26.821786 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.821753 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q4mgh\"/\"kube-root-ca.crt\"" Sep 15 12:23:26.821940 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.821813 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q4mgh\"/\"default-dockercfg-gg5x2\"" Sep 15 12:23:26.822470 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.822446 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q4mgh\"/\"openshift-service-ca.crt\"" Sep 15 12:23:26.829216 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.828831 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q4mgh/must-gather-h76n6"] Sep 15 12:23:26.990418 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.990374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53e35afe-a439-4205-8b32-67fb5609db95-must-gather-output\") pod \"must-gather-h76n6\" (UID: \"53e35afe-a439-4205-8b32-67fb5609db95\") " pod="openshift-must-gather-q4mgh/must-gather-h76n6" Sep 15 12:23:26.990418 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:26.990422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhqz\" (UniqueName: \"kubernetes.io/projected/53e35afe-a439-4205-8b32-67fb5609db95-kube-api-access-fmhqz\") pod \"must-gather-h76n6\" (UID: \"53e35afe-a439-4205-8b32-67fb5609db95\") " pod="openshift-must-gather-q4mgh/must-gather-h76n6" Sep 15 12:23:27.091566 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:27.091467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53e35afe-a439-4205-8b32-67fb5609db95-must-gather-output\") pod \"must-gather-h76n6\" (UID: \"53e35afe-a439-4205-8b32-67fb5609db95\") " pod="openshift-must-gather-q4mgh/must-gather-h76n6" Sep 15 12:23:27.091566 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:27.091513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhqz\" (UniqueName: \"kubernetes.io/projected/53e35afe-a439-4205-8b32-67fb5609db95-kube-api-access-fmhqz\") pod \"must-gather-h76n6\" (UID: \"53e35afe-a439-4205-8b32-67fb5609db95\") " pod="openshift-must-gather-q4mgh/must-gather-h76n6" Sep 15 12:23:27.091830 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:27.091813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/53e35afe-a439-4205-8b32-67fb5609db95-must-gather-output\") pod \"must-gather-h76n6\" (UID: \"53e35afe-a439-4205-8b32-67fb5609db95\") " pod="openshift-must-gather-q4mgh/must-gather-h76n6" Sep 15 12:23:27.101224 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:27.101169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhqz\" (UniqueName: \"kubernetes.io/projected/53e35afe-a439-4205-8b32-67fb5609db95-kube-api-access-fmhqz\") pod \"must-gather-h76n6\" (UID: \"53e35afe-a439-4205-8b32-67fb5609db95\") " pod="openshift-must-gather-q4mgh/must-gather-h76n6" Sep 15 12:23:27.132297 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:27.132248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q4mgh/must-gather-h76n6" Sep 15 12:23:27.406499 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:27.406474 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q4mgh/must-gather-h76n6"] Sep 15 12:23:27.409375 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:23:27.409351 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53e35afe_a439_4205_8b32_67fb5609db95.slice/crio-02a3a58a4be67dd5417d8ec3354a4fc917a46e0b4048f4ce0e11104e39079c31 WatchSource:0}: Error finding container 02a3a58a4be67dd5417d8ec3354a4fc917a46e0b4048f4ce0e11104e39079c31: Status 404 returned error can't find the container with id 02a3a58a4be67dd5417d8ec3354a4fc917a46e0b4048f4ce0e11104e39079c31 Sep 15 12:23:27.411720 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:27.411687 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 15 12:23:27.682129 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:27.682094 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q4mgh/must-gather-h76n6" event={"ID":"53e35afe-a439-4205-8b32-67fb5609db95","Type":"ContainerStarted","Data":"02a3a58a4be67dd5417d8ec3354a4fc917a46e0b4048f4ce0e11104e39079c31"} Sep 15 12:23:30.706183 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:30.706136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q4mgh/must-gather-h76n6" event={"ID":"53e35afe-a439-4205-8b32-67fb5609db95","Type":"ContainerStarted","Data":"d7bb6925a3b9ee51b1e5f4592b1381b8603c172f157c3faadbc802391e826508"} Sep 15 12:23:30.706677 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:30.706249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q4mgh/must-gather-h76n6" event={"ID":"53e35afe-a439-4205-8b32-67fb5609db95","Type":"ContainerStarted","Data":"4ae91337e40f76006b4030b45f4211231c60cce890d1e68523c12db71f80a8f3"} Sep 15 12:23:32.277996 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:32.277965 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p469s_72de5f15-c37b-4c5b-ab3b-e182a42b06ab/global-pull-secret-syncer/0.log" Sep 15 12:23:32.466503 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:32.466472 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k5k2z_1acb7e52-2368-4c5c-b3e5-c65c32136759/konnectivity-agent/0.log" Sep 15 12:23:32.591713 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:32.591628 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-16.ec2.internal_2d02dff20630896118b374c2cd0a78b5/haproxy/0.log" Sep 15 12:23:33.350106 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:33.350077 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:23:33.353485 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:33.353454 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:23:37.557782 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:37.557744 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cb98cc9fc-hsx8x_dc3fbecd-5eeb-4855-9b26-b218c74137ea/authorino/0.log" Sep 15 12:23:37.628232 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:37.628057 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-znvd5_93df8817-6a9f-4663-bb6b-2fc8d6981b97/manager/0.log" Sep 15 12:23:37.691253 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:37.691193 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-fgnnv_d069f28d-a8d6-4393-af3c-748c149d934f/manager/0.log" Sep 15 12:23:37.753647 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:37.753605 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n28h4_35bc74e8-130c-45c7-aec2-c0b72f02fb90/kuadrant-console-plugin/0.log" Sep 15 12:23:37.831278 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:37.831090 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-48pn4_ac4e1fb8-80e5-42f6-8314-4f1adb03e97a/manager/0.log" Sep 15 12:23:37.976121 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:37.976082 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-clmqc_39e7a5ff-de53-4728-81a0-d9f795c4fa47/manager/0.log" Sep 15 12:23:39.623430 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:39.623392 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-wchbc_921aeb87-2961-4759-a882-811c53e8c2da/cluster-monitoring-operator/0.log" Sep 15 12:23:39.980880 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:39.980842 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2gvq_70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9/node-exporter/0.log" Sep 15 12:23:40.024932 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:40.024905 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2gvq_70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9/kube-rbac-proxy/0.log" Sep 15 12:23:40.059308 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:40.059277 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2gvq_70178ee2-378c-4dbd-bdf6-7d45a5d4a7e9/init-textfile/0.log" Sep 15 12:23:42.838999 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:42.838911 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/2.log" Sep 15 12:23:42.848443 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:42.848404 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-hvj6z_09533bff-aa14-43ed-9121-7783f90a7eef/console-operator/3.log" Sep 15 12:23:44.776619 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.776539 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q4mgh/must-gather-h76n6" podStartSLOduration=15.857229643 podStartE2EDuration="18.776514504s" totalImagesPullingTime="2.919284861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:23:30.731352831 +0000 UTC m=+2698.076461440" watchObservedRunningTime="2026-09-15 12:23:44.776514504 +0000 UTC m=+2712.121623107" Sep 15 12:23:44.779292 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.779261 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j"] Sep 15 12:23:44.781774 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.781759 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.792255 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.792222 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j"] Sep 15 12:23:44.878719 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.878674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-podres\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.879032 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.879006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6j6k\" (UniqueName: \"kubernetes.io/projected/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-kube-api-access-p6j6k\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.879196 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.879178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-sys\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.879392 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.879375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-lib-modules\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.879555 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.879539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-proc\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981031 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.980985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-lib-modules\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.981041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-proc\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.981085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-podres\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.981133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6j6k\" (UniqueName: \"kubernetes.io/projected/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-kube-api-access-p6j6k\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.981171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-sys\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.981188 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-proc\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981241 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.981217 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-lib-modules\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981582 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.981294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-sys\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.981675 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.981652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-podres\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:44.995470 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:44.995437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6j6k\" (UniqueName: \"kubernetes.io/projected/40d361f9-2a5c-4adc-89eb-8b9bb0644abd-kube-api-access-p6j6k\") pod \"perf-node-gather-daemonset-s8t9j\" (UID: \"40d361f9-2a5c-4adc-89eb-8b9bb0644abd\") " pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:45.096274 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:45.096162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:45.392261 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:45.392232 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j"] Sep 15 12:23:45.395179 ip-10-0-140-16 kubenswrapper[2579]: W0915 12:23:45.395148 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod40d361f9_2a5c_4adc_89eb_8b9bb0644abd.slice/crio-6c1823c33b1d3587e8a528869e93ef0da7757a0460d354366931e87b6adf498c WatchSource:0}: Error finding container 6c1823c33b1d3587e8a528869e93ef0da7757a0460d354366931e87b6adf498c: Status 404 returned error can't find the container with id 6c1823c33b1d3587e8a528869e93ef0da7757a0460d354366931e87b6adf498c Sep 15 12:23:45.520852 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:45.520826 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d29g9_ddba62f7-548c-494e-841d-e5ed366f84ce/dns/0.log" Sep 15 12:23:45.553984 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:45.553952 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d29g9_ddba62f7-548c-494e-841d-e5ed366f84ce/kube-rbac-proxy/0.log" Sep 15 12:23:45.585340 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:45.585309 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2kk9v_99912355-5a7e-4bae-9df2-d1a431b0b5fe/dns-node-resolver/0.log" Sep 15 12:23:45.823508 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:45.823468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" event={"ID":"40d361f9-2a5c-4adc-89eb-8b9bb0644abd","Type":"ContainerStarted","Data":"1c1c047e937956e234f409006acbc1e1c3da41aae1fe3c4f039dc3411d35a646"} Sep 15 12:23:45.823993 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:45.823535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" event={"ID":"40d361f9-2a5c-4adc-89eb-8b9bb0644abd","Type":"ContainerStarted","Data":"6c1823c33b1d3587e8a528869e93ef0da7757a0460d354366931e87b6adf498c"} Sep 15 12:23:45.823993 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:45.823678 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:46.379908 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:46.379864 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-69tt7_840cfbe9-71a6-4d47-b285-292f5462bb62/node-ca/0.log" Sep 15 12:23:48.299476 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:48.299445 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-df8dfc748-4sgj9_efdd1b29-a854-4729-9b69-b1819af7d102/kube-auth-proxy/0.log" Sep 15 12:23:48.422245 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:48.422189 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-j2rxj_c6f5fe03-607a-4f4a-9a59-f176bae5c3cc/payload-pre-processing/0.log" Sep 15 12:23:49.566146 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:49.566113 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qr6z5_84c9ca50-20cf-40e7-a851-37f9e1488015/serve-healthcheck-canary/0.log" Sep 15 12:23:50.088956 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:50.088924 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-mxrhs_d657879d-7cf6-4d6c-82b8-71fb715059e3/insights-operator/0.log" Sep 15 12:23:50.092605 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:50.092569 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-mxrhs_d657879d-7cf6-4d6c-82b8-71fb715059e3/insights-operator/1.log" Sep 15 12:23:50.291830 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:50.291806 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jcrnr_1fda5b32-6097-4f9b-8a7d-df00f6add96a/kube-rbac-proxy/0.log" Sep 15 12:23:50.318325 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:50.318299 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jcrnr_1fda5b32-6097-4f9b-8a7d-df00f6add96a/exporter/0.log" Sep 15 12:23:50.343741 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:50.343656 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jcrnr_1fda5b32-6097-4f9b-8a7d-df00f6add96a/extractor/0.log" Sep 15 12:23:51.844229 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:51.844092 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" Sep 15 12:23:51.866542 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:51.866478 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q4mgh/perf-node-gather-daemonset-s8t9j" podStartSLOduration=7.866459205 podStartE2EDuration="7.866459205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:23:45.851859219 +0000 UTC m=+2713.196967826" watchObservedRunningTime="2026-09-15 12:23:51.866459205 +0000 UTC m=+2719.211567808" Sep 15 12:23:52.641434 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:52.641397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lknpt_621c0998-47bc-4da1-8be5-b20fdebdef73/manager/0.log" Sep 15 12:23:53.196085 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:53.196048 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-n2zwf_83dbb156-d8db-496a-8b6d-273d8b71a7d6/manager/0.log" Sep 15 12:23:53.279897 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:53.279858 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-777d95b559-z5d55_35290200-3990-4083-9f38-5452dc8b075a/manager/0.log" Sep 15 12:23:53.756455 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:53.756424 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66bc7bc78c-lpkmq_5204c2ed-93f0-40c7-a024-855959b4f0a0/manager/0.log" Sep 15 12:23:53.850966 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:53.850937 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-66bc7bc78c-lpkmq_5204c2ed-93f0-40c7-a024-855959b4f0a0/copy-manifests/0.log" Sep 15 12:23:53.973276 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:53.973234 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-df8dfc748-4sgj9_efdd1b29-a854-4729-9b69-b1819af7d102/kube-auth-proxy/0.log" Sep 15 12:23:54.064587 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:54.064505 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-j2rxj_c6f5fe03-607a-4f4a-9a59-f176bae5c3cc/payload-pre-processing/0.log" Sep 15 12:23:56.728349 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:56.728300 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-df8dfc748-4sgj9_efdd1b29-a854-4729-9b69-b1819af7d102/kube-auth-proxy/0.log" Sep 15 12:23:56.818132 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:23:56.818072 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-j2rxj_c6f5fe03-607a-4f4a-9a59-f176bae5c3cc/payload-pre-processing/0.log" Sep 15 12:24:00.969268 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:24:00.969227 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vqnhq_bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf/migrator/0.log" Sep 15 12:24:01.004604 ip-10-0-140-16 kubenswrapper[2579]: I0915 12:24:01.004556 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vqnhq_bf6d16c9-e568-4b2c-910c-c78a6ba8e3cf/graceful-termination/0.log"