Sep 09 17:14:51.608135 ip-10-0-128-235 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 17:14:51.608146 ip-10-0-128-235 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 17:14:51.608155 ip-10-0-128-235 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 17:14:51.608490 ip-10-0-128-235 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 17:15:01.856825 ip-10-0-128-235 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 17:15:01.856845 ip-10-0-128-235 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e928afd497a44ac18c48e180fa912b99 -- Sep 09 17:16:45.523017 ip-10-0-128-235 systemd[1]: Starting Kubernetes Kubelet... Sep 09 17:16:45.963174 ip-10-0-128-235 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 17:16:45.963174 ip-10-0-128-235 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 17:16:45.963174 ip-10-0-128-235 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 17:16:45.963174 ip-10-0-128-235 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 17:16:45.963174 ip-10-0-128-235 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 17:16:45.965271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.965184 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 17:16:45.970202 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970180 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970206 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970211 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970214 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970217 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970224 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970227 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970229 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970232 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970234 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970237 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970239 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970242 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:16:45.970239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970245 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970247 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970250 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970253 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970255 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970259 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970262 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970264 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970267 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970269 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970272 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970274 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970277 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970279 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970281 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970284 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970286 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970289 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970291 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970295 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:16:45.970546 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970297 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970300 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970302 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970305 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970307 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970311 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970313 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970316 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970318 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970321 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970323 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970327 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970330 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970333 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970336 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970339 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970342 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970345 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970347 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:16:45.971026 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970350 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970352 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970355 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970357 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970360 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970362 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970365 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970367 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970370 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970373 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970375 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970378 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970381 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970383 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970386 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970388 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970391 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970393 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970396 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970399 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:16:45.971485 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970401 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970403 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970406 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970409 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970412 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970415 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970417 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970421 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970425 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970428 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970430 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970433 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970436 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970439 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970443 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970447 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970449 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970452 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970454 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:16:45.971965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970458 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970461 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970463 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970466 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970468 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970472 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970474 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.970477 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971156 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971163 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971166 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971168 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971173 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971177 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971181 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971185 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971189 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971192 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971195 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:16:45.972406 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971197 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971201 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971203 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971206 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971209 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971212 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971215 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971218 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971220 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971223 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971225 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971228 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971230 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971233 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971235 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971238 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971240 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971242 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971245 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971248 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:16:45.972900 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971251 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971253 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971256 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971258 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971260 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971263 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971265 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971268 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971271 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971274 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971276 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971279 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971282 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971285 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971287 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971290 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971292 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971295 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971297 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971300 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:16:45.973474 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971302 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971305 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971307 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971310 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971312 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971316 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971319 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971322 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971325 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971328 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971330 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971333 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971336 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971338 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971340 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971343 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971345 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971348 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971350 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:16:45.973967 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971353 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971357 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971359 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971362 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971364 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971367 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971370 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971372 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971375 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971377 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971380 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971383 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971385 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971387 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971390 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971393 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971395 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971398 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971400 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:16:45.974421 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971402 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971405 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971407 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971410 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971412 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971414 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971417 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971419 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971422 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971424 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971803 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971809 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971811 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971814 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971823 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971826 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971829 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971831 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971834 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971837 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:16:45.974881 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971840 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971842 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971845 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971847 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971850 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971852 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971855 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971857 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971862 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971865 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971867 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971870 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971872 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971874 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971877 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971880 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971882 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971885 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971888 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:16:45.975350 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971891 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971893 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971896 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971898 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971900 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971903 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971905 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971908 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971916 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971918 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971921 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971923 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971926 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971929 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971931 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971934 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971936 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971940 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971942 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971945 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:16:45.975821 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971947 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971951 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971954 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971956 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971959 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971961 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971963 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971966 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971968 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971971 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971974 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971977 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971979 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971982 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971984 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971986 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971989 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971991 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971994 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971996 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:16:45.976294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.971999 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972009 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972011 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972015 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972020 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972023 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972026 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972028 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972031 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972035 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972038 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972041 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972043 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972048 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972051 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972054 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972056 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972059 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:16:45.976768 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972061 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972064 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972066 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972068 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972071 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972073 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972075 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972078 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972080 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972083 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972085 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972088 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972177 2580 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972217 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972223 2580 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972229 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972234 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972239 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972243 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972248 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972251 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 17:16:45.977192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972254 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972257 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972261 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972264 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972267 2580 flags.go:64] FLAG: --cgroup-root="" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972270 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972274 2580 flags.go:64] FLAG: --client-ca-file="" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972277 2580 flags.go:64] FLAG: --cloud-provider="external" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972280 2580 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972284 2580 flags.go:64] FLAG: --cluster-domain="" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972288 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972291 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972294 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972297 2580 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972301 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972304 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972307 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972310 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972313 2580 flags.go:64] FLAG: --contention-profiling="false" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972316 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972319 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972322 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972325 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972329 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 17:16:45.977703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972332 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972335 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972338 2580 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972341 2580 flags.go:64] FLAG: --enable-server="true" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972346 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972350 2580 flags.go:64] FLAG: --event-burst="100" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972353 2580 flags.go:64] FLAG: --event-qps="50" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972356 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972359 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972362 2580 flags.go:64] FLAG: --eviction-hard="" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972366 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972368 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972371 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972374 2580 flags.go:64] FLAG: --eviction-soft="" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972378 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972381 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972384 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972387 2580 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972390 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972393 2580 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972395 2580 flags.go:64] FLAG: --feature-gates="" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972399 2580 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972402 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972405 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972408 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972411 2580 flags.go:64] FLAG: --healthz-port="10248" Sep 09 17:16:45.978285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972415 2580 flags.go:64] FLAG: --help="false" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972418 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-235.ec2.internal" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972421 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972424 2580 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972427 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972430 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972433 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972436 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972439 2580 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972442 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972446 2580 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972449 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972452 2580 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972455 2580 flags.go:64] FLAG: --kube-reserved="" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972458 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972461 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972464 2580 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972467 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972470 2580 flags.go:64] FLAG: --lock-file="" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972473 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972477 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972480 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972485 2580 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 17:16:45.978926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972488 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972491 2580 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972494 2580 flags.go:64] FLAG: --logging-format="text" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972497 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972500 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972503 2580 flags.go:64] FLAG: --manifest-url="" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972505 2580 flags.go:64] FLAG: --manifest-url-header="" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972509 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972512 2580 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972516 2580 flags.go:64] FLAG: --max-pods="110" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972520 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972523 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972525 2580 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972528 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972531 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972534 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972537 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972544 2580 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972547 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972552 2580 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972555 2580 flags.go:64] FLAG: --pod-cidr="" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972558 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972563 2580 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972566 2580 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 17:16:45.979482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972569 2580 flags.go:64] FLAG: --pods-per-core="0" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972572 2580 flags.go:64] FLAG: --port="10250" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972574 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972577 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ee63561d4dd17aae" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972580 2580 flags.go:64] FLAG: --qos-reserved="" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972583 2580 flags.go:64] FLAG: --read-only-port="10255" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972586 2580 flags.go:64] FLAG: --register-node="true" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972589 2580 flags.go:64] FLAG: --register-with-taints="" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972593 2580 flags.go:64] FLAG: --registry-burst="10" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972596 2580 flags.go:64] FLAG: --registry-qps="5" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972599 2580 flags.go:64] FLAG: --reserved-cpus="" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972601 2580 flags.go:64] FLAG: --reserved-memory="" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972605 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972608 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972611 2580 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972614 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972617 2580 flags.go:64] FLAG: --runonce="false" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972620 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972623 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972626 2580 flags.go:64] FLAG: --seccomp-default="false" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972629 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972631 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972634 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972637 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972640 2580 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972643 2580 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 17:16:45.980070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972646 2580 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972648 2580 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972653 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972656 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972659 2580 flags.go:64] FLAG: --system-cgroups="" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972662 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972667 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972671 2580 flags.go:64] FLAG: --tls-cert-file="" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972673 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972678 2580 flags.go:64] FLAG: --tls-min-version="" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972680 2580 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972683 2580 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972686 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972689 2580 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972692 2580 flags.go:64] FLAG: --v="2" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972696 2580 flags.go:64] FLAG: --version="false" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972700 2580 flags.go:64] FLAG: --vmodule="" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972704 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.972720 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972846 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972850 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972853 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972856 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972858 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:16:45.980727 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972861 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972864 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972866 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972869 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972871 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972874 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972877 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972880 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972882 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972885 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972890 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972893 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972897 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972900 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972902 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972905 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972908 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972911 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972915 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:16:45.981386 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972919 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972922 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972925 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972927 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972930 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972932 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972935 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972937 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972940 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972942 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972945 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972947 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972950 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972953 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972955 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972958 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972961 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972964 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972967 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972969 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:16:45.982129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972972 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972974 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972977 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972981 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972984 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972988 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972991 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972994 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972997 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.972999 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973001 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973004 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973006 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973009 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973011 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973014 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973016 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973019 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973021 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973023 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:16:45.982620 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973026 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973028 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973031 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973033 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973035 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973038 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973040 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973043 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973047 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973049 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973051 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973054 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973056 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973059 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973061 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973065 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973067 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973070 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973072 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:16:45.983153 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973075 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973080 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973083 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973085 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973088 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973090 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973100 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973103 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973105 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973108 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973110 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973113 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973115 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973118 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973120 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:16:45.983607 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.973123 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.973784 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.981851 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.981877 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.982753 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.982837 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983070 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983079 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983083 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983087 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983090 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983093 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983096 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983099 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983101 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:16:45.983981 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983104 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983107 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983109 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983112 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983114 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983117 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983120 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983122 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983125 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983127 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983131 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983133 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983136 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983139 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983144 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983147 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983150 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983152 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983155 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:16:45.984349 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983158 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983161 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983164 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983168 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983171 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983174 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983177 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983181 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983185 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983188 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983191 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983193 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983196 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983199 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983202 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983205 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983208 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983210 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983213 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:16:45.984807 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983216 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983219 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983222 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983225 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983227 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983230 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983233 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983236 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983238 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983241 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983244 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983246 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983249 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983252 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983255 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983258 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983261 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983266 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983269 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983271 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:16:45.985253 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983274 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983277 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983279 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983282 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983284 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983287 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983289 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983292 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983294 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983297 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983299 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983302 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983305 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983307 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983310 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983312 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983315 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983317 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983320 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983322 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:16:45.985734 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983325 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983327 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983330 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983332 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983335 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983337 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983340 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983343 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983346 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983348 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983351 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983353 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.983358 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983465 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983469 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 17:16:45.986205 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983472 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983475 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983478 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983481 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983484 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983487 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983489 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983492 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983494 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983497 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983500 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983503 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983505 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983508 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983510 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983513 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983516 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983518 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983521 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983523 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 17:16:45.986577 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983526 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983529 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983533 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983536 2580 feature_gate.go:328] unrecognized feature gate: Example Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983538 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983541 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983543 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983546 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983548 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983551 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983553 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983556 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983558 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983561 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983563 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983566 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983568 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983571 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983573 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983576 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 17:16:45.987103 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983578 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983581 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983583 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983590 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983593 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983595 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983598 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983601 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983603 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983607 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983610 2580 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983613 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983616 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983619 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983621 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983624 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983626 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983629 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983631 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 17:16:45.987560 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983634 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983637 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983639 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983642 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983644 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983646 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983649 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983651 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983653 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983656 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983658 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983661 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983663 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983666 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983668 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983670 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983673 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983676 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983678 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983681 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 17:16:45.988016 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983683 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983686 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983689 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983691 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983693 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983696 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983698 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983701 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983703 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983707 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983723 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983726 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983729 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983732 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983735 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983738 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983741 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 17:16:45.988484 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:45.983743 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 17:16:45.988891 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.983748 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 17:16:45.988891 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.983855 2580 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 17:16:45.989456 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.989443 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 17:16:45.990527 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.990516 2580 server.go:1019] "Starting client certificate rotation" Sep 09 17:16:45.990633 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.990618 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 17:16:45.990696 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.990686 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 17:16:45.993317 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.993305 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 17:16:45.994392 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:45.994376 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 17:16:46.005351 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.005331 2580 log.go:25] "Validated CRI v1 runtime API" Sep 09 17:16:46.009427 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.009411 2580 log.go:25] "Validated CRI v1 image API" Sep 09 17:16:46.010165 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.010148 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 17:16:46.015787 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.015765 2580 fs.go:135] Filesystem UUIDs: map[20cbbdf3-9bde-451c-b155-e75f024edb31:/dev/nvme0n1p3 46421a49-ad32-4434-8cc5-872827d28984:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 09 17:16:46.015863 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.015784 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 17:16:46.017753 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.017738 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 17:16:46.020597 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.020494 2580 manager.go:216] Machine: {Timestamp:2026-09-09 17:16:46.019119981 +0000 UTC m=+0.367481446 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200541 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2353db3f31fe635b6b5c5265e3c8bc SystemUUID:ec2353db-3f31-fe63-5b6b-5c5265e3c8bc BootID:e928afd4-97a4-4ac1-8c48-e180fa912b99 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7d:0f:79:b8:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7d:0f:79:b8:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:f5:8d:1e:bf:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 17:16:46.020597 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.020590 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 17:16:46.020719 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.020657 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 17:16:46.022728 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.022689 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 17:16:46.022880 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.022733 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-235.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 17:16:46.023596 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.023587 2580 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 17:16:46.023629 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.023600 2580 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 17:16:46.023629 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.023607 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 17:16:46.023629 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.023628 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 17:16:46.025553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.025544 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 17:16:46.030125 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.030113 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 09 17:16:46.030252 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.030236 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 17:16:46.030288 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.030270 2580 kubelet.go:486] "Attempting to sync node with API server" Sep 09 17:16:46.030288 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.030287 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 17:16:46.030342 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.030301 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 17:16:46.030342 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.030315 2580 kubelet.go:398] "Adding apiserver pod source" Sep 09 17:16:46.030342 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.030325 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 17:16:46.031893 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.031850 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 17:16:46.031984 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.031910 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 17:16:46.035654 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.035637 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 17:16:46.038560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.038542 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 17:16:46.038638 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.038583 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 17:16:46.039415 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039403 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 17:16:46.039463 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039422 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 17:16:46.039463 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039427 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 17:16:46.039463 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039433 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 17:16:46.039463 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039440 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 17:16:46.039463 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039445 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 17:16:46.039587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039467 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 17:16:46.039587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039473 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 17:16:46.039587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039480 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 17:16:46.039587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039485 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 17:16:46.039587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039493 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 17:16:46.039587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.039501 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 17:16:46.040557 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.040534 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 17:16:46.040603 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.040535 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-235.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 17:16:46.041190 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.041178 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 17:16:46.041227 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.041191 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 17:16:46.044857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.044840 2580 server.go:1268] "Started kubelet" Sep 09 17:16:46.044978 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.044930 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 17:16:46.044978 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.044935 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 17:16:46.045062 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.045008 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 17:16:46.045737 ip-10-0-128-235 systemd[1]: Started Kubernetes Kubelet. Sep 09 17:16:46.046477 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.046460 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 17:16:46.046847 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.046834 2580 server.go:310] "Adding debug handlers to kubelet server" Sep 09 17:16:46.049899 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.049178 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 17:16:46.053793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.053776 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-235.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 17:16:46.054392 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.054373 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 17:16:46.054748 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.053626 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-235.ec2.internal.18d3b68fdfa34100 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-235.ec2.internal,UID:ip-10-0-128-235.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-235.ec2.internal,},FirstTimestamp:2026-09-09 17:16:46.044815616 +0000 UTC m=+0.393177080,LastTimestamp:2026-09-09 17:16:46.044815616 +0000 UTC m=+0.393177080,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-235.ec2.internal,}" Sep 09 17:16:46.055042 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.055018 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 17:16:46.055155 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.055137 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.055808 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.055788 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 17:16:46.055808 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.055797 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 17:16:46.055954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.055823 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 17:16:46.055954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.055885 2580 factory.go:55] Registering systemd factory Sep 09 17:16:46.055954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.055900 2580 factory.go:223] Registration of the systemd container factory successfully Sep 09 17:16:46.055954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.055917 2580 reconstruct.go:97] "Volume reconstruction finished" Sep 09 17:16:46.055954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.055924 2580 reconciler.go:29] "Reconciler: start to sync state" Sep 09 17:16:46.056116 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.056096 2580 factory.go:153] Registering CRI-O factory Sep 09 17:16:46.056146 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.056119 2580 factory.go:223] Registration of the crio container factory successfully Sep 09 17:16:46.056179 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.056157 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 17:16:46.056179 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.056170 2580 factory.go:103] Registering Raw factory Sep 09 17:16:46.056247 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.056181 2580 manager.go:1196] Started watching for new ooms in manager Sep 09 17:16:46.056907 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.056896 2580 manager.go:349] Starting recovery of all containers Sep 09 17:16:46.057478 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.057445 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 17:16:46.057594 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.057571 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-235.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 17:16:46.063044 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.063000 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5nzr" Sep 09 17:16:46.068579 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.068554 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h5nzr" Sep 09 17:16:46.069022 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.069004 2580 manager.go:354] Recovery completed Sep 09 17:16:46.073263 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.073249 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:16:46.074924 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.074905 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:16:46.074995 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.074936 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:16:46.074995 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.074949 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:16:46.075487 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.075472 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 17:16:46.075487 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.075487 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 17:16:46.075562 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.075479 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:16:46.075562 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.075503 2580 state_mem.go:36] "Initialized new in-memory state store" Sep 09 17:16:46.076273 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.076112 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:16:46.076273 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.076141 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:16:46.076372 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.076155 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:16:46.077532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.077519 2580 policy_none.go:49] "None policy: Start" Sep 09 17:16:46.077600 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.077536 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 17:16:46.077600 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.077548 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 17:16:46.079534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.079521 2580 policy_none.go:47] "Start" Sep 09 17:16:46.099385 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.099369 2580 manager.go:337] "Starting Device Plugin manager" Sep 09 17:16:46.099504 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.099412 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 17:16:46.099504 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.099424 2580 server.go:87] "Starting device plugin registration server" Sep 09 17:16:46.099778 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.099764 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 17:16:46.099814 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.099782 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 17:16:46.099881 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.099869 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 17:16:46.100144 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.100028 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 17:16:46.100144 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.100045 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 17:16:46.100682 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.100648 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 17:16:46.100771 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.100695 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.122687 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.122654 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 17:16:46.133173 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.123795 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 17:16:46.133173 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.123847 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 17:16:46.133173 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.123870 2580 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 17:16:46.133173 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.123968 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 17:16:46.133173 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.125867 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:16:46.200079 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.200051 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:16:46.201074 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.201057 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:16:46.201150 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.201089 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:16:46.201150 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.201101 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:16:46.201150 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.201128 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.210270 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.210253 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.210317 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.210274 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-235.ec2.internal\": node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.224737 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.224662 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal"] Sep 09 17:16:46.224828 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.224744 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:16:46.225504 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.225490 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:16:46.225563 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.225523 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:16:46.225563 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.225537 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:16:46.227832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.227820 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:16:46.227939 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.227926 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.227977 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.227952 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:16:46.228520 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.228502 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:16:46.228586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.228533 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:16:46.228586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.228547 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:16:46.228586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.228549 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:16:46.228586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.228570 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:16:46.228586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.228583 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:16:46.230853 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.230838 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.230994 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.230860 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 17:16:46.231560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.231543 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientMemory" Sep 09 17:16:46.231625 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.231574 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 17:16:46.231625 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.231588 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeHasSufficientPID" Sep 09 17:16:46.233320 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.233306 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.257241 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.257221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0492080cb16890de553e320a85f5c5aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"0492080cb16890de553e320a85f5c5aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.257324 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.257244 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0492080cb16890de553e320a85f5c5aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"0492080cb16890de553e320a85f5c5aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.257324 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.257259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4f8f7c2419a6847f0c2e2dd84ea218d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-235.ec2.internal\" (UID: \"f4f8f7c2419a6847f0c2e2dd84ea218d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.261339 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.261321 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-235.ec2.internal\" not found" node="ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.265596 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.265583 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-235.ec2.internal\" not found" node="ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.333780 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.333749 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.358122 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.358102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0492080cb16890de553e320a85f5c5aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"0492080cb16890de553e320a85f5c5aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.358244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.358134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0492080cb16890de553e320a85f5c5aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"0492080cb16890de553e320a85f5c5aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.358244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.358159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4f8f7c2419a6847f0c2e2dd84ea218d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-235.ec2.internal\" (UID: \"f4f8f7c2419a6847f0c2e2dd84ea218d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.358244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.358206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0492080cb16890de553e320a85f5c5aa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"0492080cb16890de553e320a85f5c5aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.358244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.358218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0492080cb16890de553e320a85f5c5aa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal\" (UID: \"0492080cb16890de553e320a85f5c5aa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.358244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.358207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f4f8f7c2419a6847f0c2e2dd84ea218d-config\") pod \"kube-apiserver-proxy-ip-10-0-128-235.ec2.internal\" (UID: \"f4f8f7c2419a6847f0c2e2dd84ea218d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.434053 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.434020 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.534890 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.534809 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.565244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.565219 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.569380 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.569358 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Sep 09 17:16:46.635793 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.635759 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.736232 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.736203 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.837320 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.837263 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.937984 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:46.937954 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-235.ec2.internal\" not found" Sep 09 17:16:46.991372 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.991337 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 17:16:46.991904 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:46.991501 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 17:16:47.023932 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.023907 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:16:47.031576 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.031560 2580 apiserver.go:52] "Watching apiserver" Sep 09 17:16:47.039104 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.039086 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:16:47.040237 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.040220 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 17:16:47.040546 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.040526 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-bkxw4","openshift-image-registry/node-ca-vcxxj","openshift-network-operator/iptables-alerter-lqdm7","openshift-ovn-kubernetes/ovnkube-node-z4nmc","kube-system/global-pull-secret-syncer-mt749","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq","openshift-multus/multus-additional-cni-plugins-nr8h7","openshift-multus/multus-vrk7q","openshift-multus/network-metrics-daemon-kkvpl","openshift-network-diagnostics/network-check-target-xf8st","kube-system/konnectivity-agent-vzr46","openshift-cluster-node-tuning-operator/tuned-98bh5"] Sep 09 17:16:47.043245 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.043230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.045478 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.045460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.045688 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.045667 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwkkc\"" Sep 09 17:16:47.045762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.045671 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 17:16:47.045762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.045715 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 17:16:47.048021 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.048003 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 17:16:47.048095 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.048030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.048389 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.048365 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 17:16:47.048543 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.048525 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 17:16:47.048616 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.048566 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ptp86\"" Sep 09 17:16:47.050378 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.050362 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.050476 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.050419 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:16:47.050540 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.050512 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 17:16:47.050621 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.050604 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 17:16:47.050621 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.050613 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gk86x\"" Sep 09 17:16:47.050816 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.050795 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 17:16:47.050816 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.050782 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 17:16:47.051649 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.051628 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 17:16:47.052141 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.052122 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 17:16:47.053501 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.053482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.054516 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.054497 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 17:16:47.055214 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.055197 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" Sep 09 17:16:47.056007 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.055983 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xqwr7\"" Sep 09 17:16:47.056101 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.056080 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 17:16:47.056161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.056147 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 17:16:47.056363 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.056346 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 17:16:47.056556 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.056539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.059057 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.059019 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 17:16:47.059127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.059074 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.059273 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.059254 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mwv8w\"" Sep 09 17:16:47.059408 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.059392 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 17:16:47.059408 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.059402 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 17:16:47.059529 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.059512 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 17:16:47.060298 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.060284 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 17:16:47.061463 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-slash\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.061508 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-systemd\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.061508 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061488 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:47.061597 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-etc-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.061597 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.061597 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.061546 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:16:47.061597 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061582 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.061760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07eab57a-6bbe-4919-8121-a5fb27de7f48-tmp-dir\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.061760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44fac9f1-d2e6-4397-bb60-47cf60343669-serviceca\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.061760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlkd\" (UniqueName: \"kubernetes.io/projected/44fac9f1-d2e6-4397-bb60-47cf60343669-kube-api-access-shlkd\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.061760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-kubelet\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.061760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-run-netns\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061778 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-env-overrides\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb607612-0bb9-4046-9e6f-c499d4eb1564-dbus\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-systemd-units\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-socket-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-registration-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-device-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44fac9f1-d2e6-4397-bb60-47cf60343669-host\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-cni-netd\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovnkube-config\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovn-node-metrics-cert\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.061977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcxqv\" (UniqueName: \"kubernetes.io/projected/46c0b0f4-1521-4894-a946-d3c93fb83c39-kube-api-access-tcxqv\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062001 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 17:16:47.062027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062019 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssbjj\" (UniqueName: \"kubernetes.io/projected/07eab57a-6bbe-4919-8121-a5fb27de7f48-kube-api-access-ssbjj\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-cni-bin\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062087 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vlbqv\"" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-sys-fs\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-system-cni-dir\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062158 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07eab57a-6bbe-4919-8121-a5fb27de7f48-hosts-file\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062174 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-var-lib-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-ovn\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovnkube-script-lib\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbvr\" (UniqueName: \"kubernetes.io/projected/749525cd-7749-4802-8d65-3c75a8b8461b-kube-api-access-2fbvr\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062350 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpgh\" (UniqueName: \"kubernetes.io/projected/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-kube-api-access-sfpgh\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.062535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-log-socket\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.063006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.063006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb607612-0bb9-4046-9e6f-c499d4eb1564-kubelet-config\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.063006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-node-log\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.063006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.063006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cnibin\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.063006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.062474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cni-binary-copy\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.064065 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.064050 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:47.064164 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.064146 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:16:47.066396 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.066372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:16:47.068829 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.068795 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rrsh8\"" Sep 09 17:16:47.068909 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.068879 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.068966 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.068948 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 17:16:47.069053 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.069037 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 17:16:47.069406 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.069389 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 17:16:47.071213 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.071143 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 17:11:46 +0000 UTC" deadline="2028-03-12 01:46:50.420156922 +0000 UTC" Sep 09 17:16:47.071213 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.071212 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13184h30m3.348949268s" Sep 09 17:16:47.071348 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.071266 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.071348 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.071283 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 17:16:47.071348 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.071316 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hb4b5\"" Sep 09 17:16:47.071549 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.071534 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:16:47.072296 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.072278 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 17:16:47.072372 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.072342 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" Sep 09 17:16:47.072523 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.072506 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal"] Sep 09 17:16:47.073803 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.073777 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:16:47.074074 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.074057 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 17:16:47.074170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.074125 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 17:16:47.074170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.074126 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q8xdw\"" Sep 09 17:16:47.093913 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.093859 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 17:16:47.093913 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.093869 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kvvlm" Sep 09 17:16:47.094208 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.094191 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal"] Sep 09 17:16:47.100210 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.100193 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kvvlm" Sep 09 17:16:47.156831 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.156815 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 17:16:47.163305 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-etc-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163313 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07eab57a-6bbe-4919-8121-a5fb27de7f48-tmp-dir\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.163370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22eae431-e183-4dfb-a650-967703b13aea-cni-binary-copy\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.163440 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44fac9f1-d2e6-4397-bb60-47cf60343669-serviceca\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.163440 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163415 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlkd\" (UniqueName: \"kubernetes.io/projected/44fac9f1-d2e6-4397-bb60-47cf60343669-kube-api-access-shlkd\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.163525 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-run-netns\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163525 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163415 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-etc-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163525 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb607612-0bb9-4046-9e6f-c499d4eb1564-dbus\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.163525 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-kubelet\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.163666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163560 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fb607612-0bb9-4046-9e6f-c499d4eb1564-dbus\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.163666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-run-netns\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e40de956-6de3-45a1-9117-bd61ee8a6bc1-konnectivity-ca\") pod \"konnectivity-agent-vzr46\" (UID: \"e40de956-6de3-45a1-9117-bd61ee8a6bc1\") " pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:16:47.163666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-registration-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.163666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-cni-netd\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovnkube-config\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-cni-netd\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163725 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-registration-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.163876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovn-node-metrics-cert\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcxqv\" (UniqueName: \"kubernetes.io/projected/46c0b0f4-1521-4894-a946-d3c93fb83c39-kube-api-access-tcxqv\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.163876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163830 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-conf-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.163876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-lib-modules\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44fac9f1-d2e6-4397-bb60-47cf60343669-serviceca\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.163979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-cni-bin\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164047 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-system-cni-dir\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-cni-bin\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.164098 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-system-cni-dir\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164114 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovnkube-config\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-hostroot\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-modprobe-d\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.164180 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.164172 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret podName:fb607612-0bb9-4046-9e6f-c499d4eb1564 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:47.664153548 +0000 UTC m=+2.012515002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret") pod "global-pull-secret-syncer-mt749" (UID: "fb607612-0bb9-4046-9e6f-c499d4eb1564") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164217 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164266 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-socket-dir-parent\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-log-socket\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22eae431-e183-4dfb-a650-967703b13aea-multus-daemon-config\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-run-ovn-kubernetes\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-kubernetes\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb607612-0bb9-4046-9e6f-c499d4eb1564-kubelet-config\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07eab57a-6bbe-4919-8121-a5fb27de7f48-tmp-dir\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-log-socket\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.164762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164446 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cnibin\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fb607612-0bb9-4046-9e6f-c499d4eb1564-kubelet-config\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cni-binary-copy\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164477 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cnibin\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164454 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysctl-d\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysctl-conf\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-iptables-alerter-script\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-slash\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-systemd\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164663 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-systemd\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164631 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-slash\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-cnibin\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164697 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.165631 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164701 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-multus-certs\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9v2r\" (UniqueName: \"kubernetes.io/projected/22eae431-e183-4dfb-a650-967703b13aea-kube-api-access-g9v2r\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e40de956-6de3-45a1-9117-bd61ee8a6bc1-agent-certs\") pod \"konnectivity-agent-vzr46\" (UID: \"e40de956-6de3-45a1-9117-bd61ee8a6bc1\") " pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-kubelet\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164798 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-env-overrides\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-cni-bin\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-host-kubelet\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164843 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-etc-kubernetes\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysconfig\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-systemd\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-var-lib-kubelet\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1563e128-5776-4bc3-9338-95e13c94cc62-etc-tuned\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.164984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-systemd-units\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-socket-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165012 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-systemd-units\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165029 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cni-binary-copy\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165037 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-device-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.166242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165070 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-device-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtmd\" (UniqueName: \"kubernetes.io/projected/f9649a0b-cc7d-4eb0-9952-865e10002bd7-kube-api-access-kwtmd\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165116 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-run\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-socket-dir\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95rr\" (UniqueName: \"kubernetes.io/projected/1563e128-5776-4bc3-9338-95e13c94cc62-kube-api-access-n95rr\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwfq\" (UniqueName: \"kubernetes.io/projected/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-kube-api-access-8rwfq\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165186 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-env-overrides\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44fac9f1-d2e6-4397-bb60-47cf60343669-host\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44fac9f1-d2e6-4397-bb60-47cf60343669-host\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssbjj\" (UniqueName: \"kubernetes.io/projected/07eab57a-6bbe-4919-8121-a5fb27de7f48-kube-api-access-ssbjj\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-system-cni-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-host\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1563e128-5776-4bc3-9338-95e13c94cc62-tmp\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-sys-fs\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07eab57a-6bbe-4919-8121-a5fb27de7f48-hosts-file\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165373 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-cni-multus\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.166751 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165415 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/749525cd-7749-4802-8d65-3c75a8b8461b-sys-fs\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-sys\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165430 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/07eab57a-6bbe-4919-8121-a5fb27de7f48-hosts-file\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165455 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-var-lib-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165482 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-ovn\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovnkube-script-lib\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-run-ovn\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbvr\" (UniqueName: \"kubernetes.io/projected/749525cd-7749-4802-8d65-3c75a8b8461b-kube-api-access-2fbvr\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpgh\" (UniqueName: \"kubernetes.io/projected/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-kube-api-access-sfpgh\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-k8s-cni-cncf-io\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165654 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-node-log\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-var-lib-openvswitch\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-cni-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-netns\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46c0b0f4-1521-4894-a946-d3c93fb83c39-node-log\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.167282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-host-slash\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.167755 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.165938 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovnkube-script-lib\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.167755 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.166007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.167755 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.167609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46c0b0f4-1521-4894-a946-d3c93fb83c39-ovn-node-metrics-cert\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.172780 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.172686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlkd\" (UniqueName: \"kubernetes.io/projected/44fac9f1-d2e6-4397-bb60-47cf60343669-kube-api-access-shlkd\") pod \"node-ca-vcxxj\" (UID: \"44fac9f1-d2e6-4397-bb60-47cf60343669\") " pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.173519 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.173486 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcxqv\" (UniqueName: \"kubernetes.io/projected/46c0b0f4-1521-4894-a946-d3c93fb83c39-kube-api-access-tcxqv\") pod \"ovnkube-node-z4nmc\" (UID: \"46c0b0f4-1521-4894-a946-d3c93fb83c39\") " pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.173722 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.173696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpgh\" (UniqueName: \"kubernetes.io/projected/9bf0bee3-0ee6-4c63-b165-87ddd060ec71-kube-api-access-sfpgh\") pod \"multus-additional-cni-plugins-nr8h7\" (UID: \"9bf0bee3-0ee6-4c63-b165-87ddd060ec71\") " pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.173849 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.173783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssbjj\" (UniqueName: \"kubernetes.io/projected/07eab57a-6bbe-4919-8121-a5fb27de7f48-kube-api-access-ssbjj\") pod \"node-resolver-bkxw4\" (UID: \"07eab57a-6bbe-4919-8121-a5fb27de7f48\") " pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.174147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.174119 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbvr\" (UniqueName: \"kubernetes.io/projected/749525cd-7749-4802-8d65-3c75a8b8461b-kube-api-access-2fbvr\") pod \"aws-ebs-csi-driver-node-dfdmq\" (UID: \"749525cd-7749-4802-8d65-3c75a8b8461b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.267068 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-socket-dir-parent\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267068 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267069 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22eae431-e183-4dfb-a650-967703b13aea-multus-daemon-config\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-kubernetes\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.267211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:47.267211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-socket-dir-parent\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267180 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysctl-d\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.267211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysctl-conf\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267229 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-iptables-alerter-script\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-cnibin\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267265 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-kubernetes\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-multus-certs\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267304 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9v2r\" (UniqueName: \"kubernetes.io/projected/22eae431-e183-4dfb-a650-967703b13aea-kube-api-access-g9v2r\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267318 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-cnibin\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267329 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e40de956-6de3-45a1-9117-bd61ee8a6bc1-agent-certs\") pod \"konnectivity-agent-vzr46\" (UID: \"e40de956-6de3-45a1-9117-bd61ee8a6bc1\") " pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysctl-d\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267354 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-cni-bin\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-multus-certs\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysctl-conf\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-etc-kubernetes\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-cni-bin\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267405 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-etc-kubernetes\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysconfig\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-systemd\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.267441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267446 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-sysconfig\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-var-lib-kubelet\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267486 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-var-lib-kubelet\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1563e128-5776-4bc3-9338-95e13c94cc62-etc-tuned\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-systemd\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtmd\" (UniqueName: \"kubernetes.io/projected/f9649a0b-cc7d-4eb0-9952-865e10002bd7-kube-api-access-kwtmd\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-run\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95rr\" (UniqueName: \"kubernetes.io/projected/1563e128-5776-4bc3-9338-95e13c94cc62-kube-api-access-n95rr\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwfq\" (UniqueName: \"kubernetes.io/projected/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-kube-api-access-8rwfq\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-system-cni-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267603 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-run\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267610 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-host\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1563e128-5776-4bc3-9338-95e13c94cc62-tmp\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-cni-multus\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267661 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-system-cni-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-sys\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-k8s-cni-cncf-io\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.268272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-cni-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267726 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-cni-multus\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-netns\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-host-slash\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-sys\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22eae431-e183-4dfb-a650-967703b13aea-cni-binary-copy\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-host\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-kubelet\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267874 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e40de956-6de3-45a1-9117-bd61ee8a6bc1-konnectivity-ca\") pod \"konnectivity-agent-vzr46\" (UID: \"e40de956-6de3-45a1-9117-bd61ee8a6bc1\") " pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.267884 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267891 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-host-slash\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-conf-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-cni-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.267945 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:47.767928075 +0000 UTC m=+2.116289529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267949 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-multus-conf-dir\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-lib-modules\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-var-lib-kubelet\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.267962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-k8s-cni-cncf-io\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.269239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-hostroot\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-modprobe-d\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-etc-modprobe-d\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-hostroot\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268171 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1563e128-5776-4bc3-9338-95e13c94cc62-lib-modules\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22eae431-e183-4dfb-a650-967703b13aea-host-run-netns\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268390 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/22eae431-e183-4dfb-a650-967703b13aea-multus-daemon-config\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268535 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e40de956-6de3-45a1-9117-bd61ee8a6bc1-konnectivity-ca\") pod \"konnectivity-agent-vzr46\" (UID: \"e40de956-6de3-45a1-9117-bd61ee8a6bc1\") " pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-iptables-alerter-script\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.268816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22eae431-e183-4dfb-a650-967703b13aea-cni-binary-copy\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.269692 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1563e128-5776-4bc3-9338-95e13c94cc62-etc-tuned\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.270181 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.269848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e40de956-6de3-45a1-9117-bd61ee8a6bc1-agent-certs\") pod \"konnectivity-agent-vzr46\" (UID: \"e40de956-6de3-45a1-9117-bd61ee8a6bc1\") " pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:16:47.270512 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.270310 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1563e128-5776-4bc3-9338-95e13c94cc62-tmp\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.275466 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.275444 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:16:47.275577 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.275470 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:16:47.275577 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.275484 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5zq9k for pod openshift-network-diagnostics/network-check-target-xf8st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:47.275577 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.275549 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k podName:586fc5a3-1e20-4b14-8380-86ca5ecb6e10 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:47.775533053 +0000 UTC m=+2.123894507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5zq9k" (UniqueName: "kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k") pod "network-check-target-xf8st" (UID: "586fc5a3-1e20-4b14-8380-86ca5ecb6e10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:47.277204 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.277178 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtmd\" (UniqueName: \"kubernetes.io/projected/f9649a0b-cc7d-4eb0-9952-865e10002bd7-kube-api-access-kwtmd\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:47.277386 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.277367 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwfq\" (UniqueName: \"kubernetes.io/projected/0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9-kube-api-access-8rwfq\") pod \"iptables-alerter-lqdm7\" (UID: \"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9\") " pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.277459 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.277442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9v2r\" (UniqueName: \"kubernetes.io/projected/22eae431-e183-4dfb-a650-967703b13aea-kube-api-access-g9v2r\") pod \"multus-vrk7q\" (UID: \"22eae431-e183-4dfb-a650-967703b13aea\") " pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.278000 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.277979 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95rr\" (UniqueName: \"kubernetes.io/projected/1563e128-5776-4bc3-9338-95e13c94cc62-kube-api-access-n95rr\") pod \"tuned-98bh5\" (UID: \"1563e128-5776-4bc3-9338-95e13c94cc62\") " pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.373196 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.373132 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bkxw4" Sep 09 17:16:47.399473 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.399439 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vcxxj" Sep 09 17:16:47.424372 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.424341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:16:47.444130 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.444102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" Sep 09 17:16:47.450261 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.450242 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" Sep 09 17:16:47.460800 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.460771 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vrk7q" Sep 09 17:16:47.483213 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.483196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:16:47.501327 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.501309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-98bh5" Sep 09 17:16:47.506193 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.506172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lqdm7" Sep 09 17:16:47.509185 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.509156 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf0bee3_0ee6_4c63_b165_87ddd060ec71.slice/crio-eb98bdd06aea1418141496fd8a7626f5df2e892980953ccc8c674789f29917b1 WatchSource:0}: Error finding container eb98bdd06aea1418141496fd8a7626f5df2e892980953ccc8c674789f29917b1: Status 404 returned error can't find the container with id eb98bdd06aea1418141496fd8a7626f5df2e892980953ccc8c674789f29917b1 Sep 09 17:16:47.509403 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.509378 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40de956_6de3_45a1_9117_bd61ee8a6bc1.slice/crio-27b5487a245517c49e80b74885dd9d429aca55cdb26b52302f90bba61836687b WatchSource:0}: Error finding container 27b5487a245517c49e80b74885dd9d429aca55cdb26b52302f90bba61836687b: Status 404 returned error can't find the container with id 27b5487a245517c49e80b74885dd9d429aca55cdb26b52302f90bba61836687b Sep 09 17:16:47.510129 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.510099 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22eae431_e183_4dfb_a650_967703b13aea.slice/crio-0a447ae88ef652402d721d12bd3b588c998a7b36b516d702b95e006c177112de WatchSource:0}: Error finding container 0a447ae88ef652402d721d12bd3b588c998a7b36b516d702b95e006c177112de: Status 404 returned error can't find the container with id 0a447ae88ef652402d721d12bd3b588c998a7b36b516d702b95e006c177112de Sep 09 17:16:47.510930 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.510913 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0492080cb16890de553e320a85f5c5aa.slice/crio-aff1dfadbf6de79cb0a3eb17972b2194a0896e72c6f45270fca57f387a19b649 WatchSource:0}: Error finding container aff1dfadbf6de79cb0a3eb17972b2194a0896e72c6f45270fca57f387a19b649: Status 404 returned error can't find the container with id aff1dfadbf6de79cb0a3eb17972b2194a0896e72c6f45270fca57f387a19b649 Sep 09 17:16:47.512965 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.512790 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f8f7c2419a6847f0c2e2dd84ea218d.slice/crio-643ecbccc9ea2bb1521647ca5b02567c79be900fb325f4673d1498c90c2ea30e WatchSource:0}: Error finding container 643ecbccc9ea2bb1521647ca5b02567c79be900fb325f4673d1498c90c2ea30e: Status 404 returned error can't find the container with id 643ecbccc9ea2bb1521647ca5b02567c79be900fb325f4673d1498c90c2ea30e Sep 09 17:16:47.515143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.515054 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:16:47.515914 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.515807 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1563e128_5776_4bc3_9338_95e13c94cc62.slice/crio-bed639e715fecc3efb1a3dfee899cca32b79956eb6a576da8cfaadba6f13446b WatchSource:0}: Error finding container bed639e715fecc3efb1a3dfee899cca32b79956eb6a576da8cfaadba6f13446b: Status 404 returned error can't find the container with id bed639e715fecc3efb1a3dfee899cca32b79956eb6a576da8cfaadba6f13446b Sep 09 17:16:47.519124 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.519104 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b41bc3d_65ad_475d_ae22_1ce0d02e6eb9.slice/crio-b995c4fb907775f12826f11bb469b89f1fff144d1c5507f9883f05dc7fbf120a WatchSource:0}: Error finding container b995c4fb907775f12826f11bb469b89f1fff144d1c5507f9883f05dc7fbf120a: Status 404 returned error can't find the container with id b995c4fb907775f12826f11bb469b89f1fff144d1c5507f9883f05dc7fbf120a Sep 09 17:16:47.571586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.571561 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:16:47.671669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.671590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:47.671812 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.671751 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:47.671812 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.671807 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret podName:fb607612-0bb9-4046-9e6f-c499d4eb1564 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:48.671792372 +0000 UTC m=+3.020153824 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret") pod "global-pull-secret-syncer-mt749" (UID: "fb607612-0bb9-4046-9e6f-c499d4eb1564") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:47.772540 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.772506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:47.772752 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.772642 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:47.772752 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.772713 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:48.772691667 +0000 UTC m=+3.121053121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:47.802721 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.802693 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 17:16:47.806239 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.806214 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749525cd_7749_4802_8d65_3c75a8b8461b.slice/crio-de73f6e87d22b475928150cb91ee974186b5ad35adf267389c2bc47196180b5d WatchSource:0}: Error finding container de73f6e87d22b475928150cb91ee974186b5ad35adf267389c2bc47196180b5d: Status 404 returned error can't find the container with id de73f6e87d22b475928150cb91ee974186b5ad35adf267389c2bc47196180b5d Sep 09 17:16:47.873055 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:47.873015 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:47.873200 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.873169 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:16:47.873200 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.873186 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:16:47.873200 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.873196 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5zq9k for pod openshift-network-diagnostics/network-check-target-xf8st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:47.873329 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:47.873250 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k podName:586fc5a3-1e20-4b14-8380-86ca5ecb6e10 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:48.873231815 +0000 UTC m=+3.221593289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zq9k" (UniqueName: "kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k") pod "network-check-target-xf8st" (UID: "586fc5a3-1e20-4b14-8380-86ca5ecb6e10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:47.880672 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.880640 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fac9f1_d2e6_4397_bb60_47cf60343669.slice/crio-d37b69ae094790ce4637606418fde694a047246ced09967f6ea6455bdd039fe3 WatchSource:0}: Error finding container d37b69ae094790ce4637606418fde694a047246ced09967f6ea6455bdd039fe3: Status 404 returned error can't find the container with id d37b69ae094790ce4637606418fde694a047246ced09967f6ea6455bdd039fe3 Sep 09 17:16:47.913670 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:47.913633 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c0b0f4_1521_4894_a946_d3c93fb83c39.slice/crio-29007f37cfb46c3103ab247eec0edb4335b45c285f57af08ede1d5f7efacf209 WatchSource:0}: Error finding container 29007f37cfb46c3103ab247eec0edb4335b45c285f57af08ede1d5f7efacf209: Status 404 returned error can't find the container with id 29007f37cfb46c3103ab247eec0edb4335b45c285f57af08ede1d5f7efacf209 Sep 09 17:16:48.016605 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:16:48.016573 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07eab57a_6bbe_4919_8121_a5fb27de7f48.slice/crio-2b3a6d651aecb138a9de1ac086774de122987545985dfc529fc9f11196c94c71 WatchSource:0}: Error finding container 2b3a6d651aecb138a9de1ac086774de122987545985dfc529fc9f11196c94c71: Status 404 returned error can't find the container with id 2b3a6d651aecb138a9de1ac086774de122987545985dfc529fc9f11196c94c71 Sep 09 17:16:48.104025 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.103960 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 17:11:47 +0000 UTC" deadline="2028-03-18 08:33:43.251097494 +0000 UTC" Sep 09 17:16:48.104025 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.104017 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13335h16m55.14708333s" Sep 09 17:16:48.128651 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.128607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" event={"ID":"0492080cb16890de553e320a85f5c5aa","Type":"ContainerStarted","Data":"aff1dfadbf6de79cb0a3eb17972b2194a0896e72c6f45270fca57f387a19b649"} Sep 09 17:16:48.129638 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.129617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bkxw4" event={"ID":"07eab57a-6bbe-4919-8121-a5fb27de7f48","Type":"ContainerStarted","Data":"2b3a6d651aecb138a9de1ac086774de122987545985dfc529fc9f11196c94c71"} Sep 09 17:16:48.130628 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.130610 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-98bh5" event={"ID":"1563e128-5776-4bc3-9338-95e13c94cc62","Type":"ContainerStarted","Data":"bed639e715fecc3efb1a3dfee899cca32b79956eb6a576da8cfaadba6f13446b"} Sep 09 17:16:48.131630 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.131612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vzr46" event={"ID":"e40de956-6de3-45a1-9117-bd61ee8a6bc1","Type":"ContainerStarted","Data":"27b5487a245517c49e80b74885dd9d429aca55cdb26b52302f90bba61836687b"} Sep 09 17:16:48.133457 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.133438 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" event={"ID":"9bf0bee3-0ee6-4c63-b165-87ddd060ec71","Type":"ContainerStarted","Data":"eb98bdd06aea1418141496fd8a7626f5df2e892980953ccc8c674789f29917b1"} Sep 09 17:16:48.134553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.134530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"29007f37cfb46c3103ab247eec0edb4335b45c285f57af08ede1d5f7efacf209"} Sep 09 17:16:48.137531 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.137507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vcxxj" event={"ID":"44fac9f1-d2e6-4397-bb60-47cf60343669","Type":"ContainerStarted","Data":"d37b69ae094790ce4637606418fde694a047246ced09967f6ea6455bdd039fe3"} Sep 09 17:16:48.138618 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.138598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" event={"ID":"749525cd-7749-4802-8d65-3c75a8b8461b","Type":"ContainerStarted","Data":"de73f6e87d22b475928150cb91ee974186b5ad35adf267389c2bc47196180b5d"} Sep 09 17:16:48.139634 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.139616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqdm7" event={"ID":"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9","Type":"ContainerStarted","Data":"b995c4fb907775f12826f11bb469b89f1fff144d1c5507f9883f05dc7fbf120a"} Sep 09 17:16:48.140595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.140575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" event={"ID":"f4f8f7c2419a6847f0c2e2dd84ea218d","Type":"ContainerStarted","Data":"643ecbccc9ea2bb1521647ca5b02567c79be900fb325f4673d1498c90c2ea30e"} Sep 09 17:16:48.141671 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.141652 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vrk7q" event={"ID":"22eae431-e183-4dfb-a650-967703b13aea","Type":"ContainerStarted","Data":"0a447ae88ef652402d721d12bd3b588c998a7b36b516d702b95e006c177112de"} Sep 09 17:16:48.681166 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.681129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:48.681453 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:48.681434 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:48.681530 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:48.681504 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret podName:fb607612-0bb9-4046-9e6f-c499d4eb1564 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:50.681484019 +0000 UTC m=+5.029845487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret") pod "global-pull-secret-syncer-mt749" (UID: "fb607612-0bb9-4046-9e6f-c499d4eb1564") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:48.781954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.781916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:48.782191 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:48.782138 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:48.782279 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:48.782202 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:50.782183478 +0000 UTC m=+5.130544949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:48.882977 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:48.882935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:48.883167 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:48.883126 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:16:48.883167 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:48.883152 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:16:48.883167 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:48.883168 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5zq9k for pod openshift-network-diagnostics/network-check-target-xf8st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:48.883356 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:48.883232 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k podName:586fc5a3-1e20-4b14-8380-86ca5ecb6e10 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:50.883212844 +0000 UTC m=+5.231574316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zq9k" (UniqueName: "kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k") pod "network-check-target-xf8st" (UID: "586fc5a3-1e20-4b14-8380-86ca5ecb6e10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:49.104308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:49.104160 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 17:11:47 +0000 UTC" deadline="2028-02-26 08:04:24.442524473 +0000 UTC" Sep 09 17:16:49.104308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:49.104225 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12830h47m35.33830386s" Sep 09 17:16:49.125726 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:49.125224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:49.125726 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:49.125350 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:16:49.125726 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:49.125498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:49.125726 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:49.125507 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:49.125726 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:49.125612 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:16:49.125726 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:49.125720 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:16:50.705746 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:50.705461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:50.706343 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:50.705745 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:50.706343 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:50.705827 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret podName:fb607612-0bb9-4046-9e6f-c499d4eb1564 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:54.705806058 +0000 UTC m=+9.054167635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret") pod "global-pull-secret-syncer-mt749" (UID: "fb607612-0bb9-4046-9e6f-c499d4eb1564") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:50.806879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:50.806757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:50.807056 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:50.806967 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:50.807056 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:50.807034 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:54.807014153 +0000 UTC m=+9.155375610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:50.907976 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:50.907936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:50.908233 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:50.908211 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:16:50.908314 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:50.908242 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:16:50.908314 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:50.908255 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5zq9k for pod openshift-network-diagnostics/network-check-target-xf8st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:50.908431 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:50.908319 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k podName:586fc5a3-1e20-4b14-8380-86ca5ecb6e10 nodeName:}" failed. No retries permitted until 2026-09-09 17:16:54.908296266 +0000 UTC m=+9.256657726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zq9k" (UniqueName: "kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k") pod "network-check-target-xf8st" (UID: "586fc5a3-1e20-4b14-8380-86ca5ecb6e10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:51.124995 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:51.124914 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:51.125175 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:51.125051 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:16:51.125471 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:51.125447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:51.125594 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:51.125565 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:16:51.125685 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:51.125614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:51.125742 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:51.125685 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:16:53.125653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:53.125156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:53.125653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:53.125201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:53.125653 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:53.125293 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:16:53.125653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:53.125170 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:53.125653 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:53.125450 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:16:53.125653 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:53.125532 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:16:54.738092 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:54.738050 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:54.738510 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:54.738212 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:54.738510 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:54.738295 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret podName:fb607612-0bb9-4046-9e6f-c499d4eb1564 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:02.738274155 +0000 UTC m=+17.086635614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret") pod "global-pull-secret-syncer-mt749" (UID: "fb607612-0bb9-4046-9e6f-c499d4eb1564") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:16:54.838578 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:54.838466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:54.838758 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:54.838608 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:54.838758 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:54.838674 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:02.838654501 +0000 UTC m=+17.187015972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:16:54.939895 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:54.939811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:54.940051 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:54.939999 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:16:54.940051 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:54.940020 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:16:54.940051 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:54.940032 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5zq9k for pod openshift-network-diagnostics/network-check-target-xf8st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:54.940205 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:54.940098 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k podName:586fc5a3-1e20-4b14-8380-86ca5ecb6e10 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:02.940079508 +0000 UTC m=+17.288440959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zq9k" (UniqueName: "kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k") pod "network-check-target-xf8st" (UID: "586fc5a3-1e20-4b14-8380-86ca5ecb6e10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:16:55.124421 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:55.124381 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:55.124591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:55.124505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:55.124591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:55.124547 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:55.124697 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:55.124514 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:16:55.124697 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:55.124639 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:16:55.124816 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:55.124698 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:16:57.124660 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:57.124583 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:57.124660 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:57.124624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:57.125229 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:57.124737 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:16:57.125229 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:57.124844 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:16:57.125229 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:57.124886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:57.125229 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:57.124949 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:16:59.125175 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:59.125139 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:16:59.125599 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:59.125139 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:16:59.125599 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:59.125246 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:16:59.125599 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:16:59.125139 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:16:59.125599 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:59.125444 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:16:59.125599 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:16:59.125362 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:01.124939 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:01.124908 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:01.125351 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:01.124915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:01.125351 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:01.125008 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:01.125351 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:01.124914 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:01.125351 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:01.125096 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:01.125351 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:01.125212 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:02.795606 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:02.795571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:02.796068 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:02.795702 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:17:02.796068 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:02.795782 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret podName:fb607612-0bb9-4046-9e6f-c499d4eb1564 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:18.795763174 +0000 UTC m=+33.144124631 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret") pod "global-pull-secret-syncer-mt749" (UID: "fb607612-0bb9-4046-9e6f-c499d4eb1564") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:17:02.896262 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:02.896232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:02.896450 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:02.896383 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:17:02.896510 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:02.896459 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:18.896438009 +0000 UTC m=+33.244799500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:17:02.998127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:02.998093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:02.998299 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:02.998279 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:17:02.998342 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:02.998309 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:17:02.998342 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:02.998321 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5zq9k for pod openshift-network-diagnostics/network-check-target-xf8st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:17:02.998411 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:02.998382 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k podName:586fc5a3-1e20-4b14-8380-86ca5ecb6e10 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:18.998363814 +0000 UTC m=+33.346725285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zq9k" (UniqueName: "kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k") pod "network-check-target-xf8st" (UID: "586fc5a3-1e20-4b14-8380-86ca5ecb6e10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:17:03.124888 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:03.124797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:03.125059 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:03.124808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:03.125059 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:03.124932 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:03.125059 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:03.124811 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:03.125059 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:03.125020 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:03.125235 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:03.125106 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:05.124535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:05.124502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:05.125049 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:05.124543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:05.125049 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:05.124510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:05.125049 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:05.124604 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:05.125049 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:05.124731 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:05.125049 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:05.124830 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:07.125653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.124857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:07.125653 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:07.125204 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:07.125653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.125300 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:07.125653 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:07.125387 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:07.125653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.125444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:07.125653 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:07.125521 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:07.198689 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.198654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-98bh5" event={"ID":"1563e128-5776-4bc3-9338-95e13c94cc62","Type":"ContainerStarted","Data":"08497ebac87e156c71aafa47a549f2ddfea593123c0fbc0a72b7bcca44520610"} Sep 09 17:17:07.201995 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.201963 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:17:07.202695 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.202627 2580 generic.go:356] "Generic (PLEG): container finished" podID="46c0b0f4-1521-4894-a946-d3c93fb83c39" containerID="88bdfbb030f2182f6cba7aefa49c0ea5c7a2788dcc62f94d3167cd637c7f8776" exitCode=1 Sep 09 17:17:07.202809 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.202695 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"e2ee70a804ef99789062b909c4421b7e62bfdb3a9f784e1a645e930bae8c9039"} Sep 09 17:17:07.202809 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.202773 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerDied","Data":"88bdfbb030f2182f6cba7aefa49c0ea5c7a2788dcc62f94d3167cd637c7f8776"} Sep 09 17:17:07.202809 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.202806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"9d861265229e5d1bc836dcf3e7cc92333891fb7949e24baaa2d704706a1e7d9e"} Sep 09 17:17:07.206314 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.205973 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" event={"ID":"f4f8f7c2419a6847f0c2e2dd84ea218d","Type":"ContainerStarted","Data":"3c4e8f1d1b83171e7c02ef4772890179b32eea8e6582cc28f9b56b5b5d857a41"} Sep 09 17:17:07.209522 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.209494 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vrk7q" event={"ID":"22eae431-e183-4dfb-a650-967703b13aea","Type":"ContainerStarted","Data":"bee7c3044bfdec1607c5062ba2580d227a2bc74c925e0f38b43b2881f9543aa6"} Sep 09 17:17:07.217758 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.217617 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-98bh5" podStartSLOduration=2.011310713 podStartE2EDuration="21.217605653s" totalImagesPullingTime="19.20629494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:07.217245925 +0000 UTC m=+21.565607398" watchObservedRunningTime="2026-09-09 17:17:07.217605653 +0000 UTC m=+21.565967160" Sep 09 17:17:07.231620 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:07.231298 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-235.ec2.internal" podStartSLOduration=20.231277617 podStartE2EDuration="20.231277617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:07.231020301 +0000 UTC m=+21.579381774" watchObservedRunningTime="2026-09-09 17:17:07.231277617 +0000 UTC m=+21.579639091" Sep 09 17:17:08.214051 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.213848 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bf0bee3-0ee6-4c63-b165-87ddd060ec71" containerID="cc3b8361f1294c19245ab5f0f84db43e1bbbac92ff42fbaa5366745ea1216b7e" exitCode=0 Sep 09 17:17:08.214569 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.213949 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" event={"ID":"9bf0bee3-0ee6-4c63-b165-87ddd060ec71","Type":"ContainerDied","Data":"cc3b8361f1294c19245ab5f0f84db43e1bbbac92ff42fbaa5366745ea1216b7e"} Sep 09 17:17:08.217486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.217465 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:17:08.217955 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.217923 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"cda0a79a1e15867f6b12e49b75a5034ca7a7c895b0965ec0a6a9882ca5c35290"} Sep 09 17:17:08.218115 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.218018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"8bbc3baee8954368fb821c810cafabdfa6ad7e80af4a9938b7f5703630a0330d"} Sep 09 17:17:08.218115 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.218068 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"e2381158d8ce469cde760a24ec32b04bdd5bd16f25458430e22e1d33694e23c4"} Sep 09 17:17:08.219510 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.219484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vcxxj" event={"ID":"44fac9f1-d2e6-4397-bb60-47cf60343669","Type":"ContainerStarted","Data":"96e41ffd9ff421a00c23d4ad36d387318b81d655782ab27989810dcdcb87a1ea"} Sep 09 17:17:08.221352 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.221327 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" event={"ID":"749525cd-7749-4802-8d65-3c75a8b8461b","Type":"ContainerStarted","Data":"83be4d5c2aa14739597fad2aabfb8ce7dab7fd6ba4ed8b356ccc946f32245af5"} Sep 09 17:17:08.223095 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.223073 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lqdm7" event={"ID":"0b41bc3d-65ad-475d-ae22-1ce0d02e6eb9","Type":"ContainerStarted","Data":"75fdff74dcd5191d228668f4947198e3a4011d2023e34d2a4591eefc95b3e5eb"} Sep 09 17:17:08.224783 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.224764 2580 generic.go:356] "Generic (PLEG): container finished" podID="0492080cb16890de553e320a85f5c5aa" containerID="ccf5758427e47c85665b29cc7acaace63bb29f6939624ecb0e69b6adefa9e609" exitCode=0 Sep 09 17:17:08.224874 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.224823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" event={"ID":"0492080cb16890de553e320a85f5c5aa","Type":"ContainerDied","Data":"ccf5758427e47c85665b29cc7acaace63bb29f6939624ecb0e69b6adefa9e609"} Sep 09 17:17:08.226560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.226536 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bkxw4" event={"ID":"07eab57a-6bbe-4919-8121-a5fb27de7f48","Type":"ContainerStarted","Data":"b83a768507be6b56bed4df84350a269275a4518e950fecc4d9b15d27e7cb1655"} Sep 09 17:17:08.228215 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.228181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vzr46" event={"ID":"e40de956-6de3-45a1-9117-bd61ee8a6bc1","Type":"ContainerStarted","Data":"24145c17fe1816c13e3529c6c3d6cfb2cbed6e67b12e7aa6fd528444baf81dfb"} Sep 09 17:17:08.236097 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.236061 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vrk7q" podStartSLOduration=2.832744128 podStartE2EDuration="22.236048244s" totalImagesPullingTime="19.403304116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:07.256606346 +0000 UTC m=+21.604967820" watchObservedRunningTime="2026-09-09 17:17:08.236048244 +0000 UTC m=+22.584409716" Sep 09 17:17:08.260825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.260758 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bkxw4" podStartSLOduration=3.555114977 podStartE2EDuration="22.260739262s" totalImagesPullingTime="18.705624285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:08.260303413 +0000 UTC m=+22.608664884" watchObservedRunningTime="2026-09-09 17:17:08.260739262 +0000 UTC m=+22.609100736" Sep 09 17:17:08.274295 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.274245 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lqdm7" podStartSLOduration=2.074290451 podStartE2EDuration="21.27422698s" totalImagesPullingTime="19.199936529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:08.274083344 +0000 UTC m=+22.622444835" watchObservedRunningTime="2026-09-09 17:17:08.27422698 +0000 UTC m=+22.622588454" Sep 09 17:17:08.288805 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.288754 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vcxxj" podStartSLOduration=3.474623804 podStartE2EDuration="22.28873672s" totalImagesPullingTime="18.814112916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:08.288461092 +0000 UTC m=+22.636822566" watchObservedRunningTime="2026-09-09 17:17:08.28873672 +0000 UTC m=+22.637098191" Sep 09 17:17:08.302451 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:08.302412 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vzr46" podStartSLOduration=3.093725735 podStartE2EDuration="22.30239858s" totalImagesPullingTime="19.208672845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:08.302218633 +0000 UTC m=+22.650580116" watchObservedRunningTime="2026-09-09 17:17:08.30239858 +0000 UTC m=+22.650760087" Sep 09 17:17:09.025028 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.024998 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 17:17:09.113318 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.113171 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T17:17:09.025022241Z","UUID":"2b53f41d-ee1d-4866-b441-a61729f29912","Handler":null,"Name":"","Endpoint":""} Sep 09 17:17:09.114807 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.114786 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 17:17:09.114941 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.114820 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 17:17:09.124453 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.124430 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:09.124577 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.124461 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:09.124629 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.124581 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:09.124629 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:09.124589 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:09.124741 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:09.124661 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:09.124785 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:09.124750 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:09.232594 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.232507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" event={"ID":"749525cd-7749-4802-8d65-3c75a8b8461b","Type":"ContainerStarted","Data":"0b04ee250e804d84100279daf38db3fe5923b14cb970d462e5455683d9da30d6"} Sep 09 17:17:09.234640 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.234612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" event={"ID":"0492080cb16890de553e320a85f5c5aa","Type":"ContainerStarted","Data":"ae9b533c933032299a37a80b3b7ba45ae4b16c5e6a852f0dd26f3aa670c169b2"} Sep 09 17:17:09.250020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:09.249977 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-235.ec2.internal" podStartSLOduration=22.249964528 podStartE2EDuration="22.249964528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:09.249387323 +0000 UTC m=+23.597748797" watchObservedRunningTime="2026-09-09 17:17:09.249964528 +0000 UTC m=+23.598326000" Sep 09 17:17:10.239350 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:10.239321 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:17:10.239845 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:10.239723 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"323ae012e8a3cd08c9f8d3e192bd627f8ca8f8fe4494351d8bb35512d5a17331"} Sep 09 17:17:10.807039 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:10.807008 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:17:10.807641 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:10.807613 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:17:11.125126 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:11.125038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:11.125126 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:11.125075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:11.125126 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:11.125075 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:11.125392 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:11.125166 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:11.125392 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:11.125256 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:11.125392 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:11.125305 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:11.244303 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:11.244261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" event={"ID":"749525cd-7749-4802-8d65-3c75a8b8461b","Type":"ContainerStarted","Data":"260c7a0a74a01549d2c3e682aafcc6690cf1c7b3e12865fdb51724b50c6b841e"} Sep 09 17:17:11.260981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:11.260905 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfdmq" podStartSLOduration=3.566993172 podStartE2EDuration="25.260841047s" totalImagesPullingTime="21.693847875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:11.2599325 +0000 UTC m=+25.608293995" watchObservedRunningTime="2026-09-09 17:17:11.260841047 +0000 UTC m=+25.609202521" Sep 09 17:17:12.250895 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:12.250854 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:17:12.251396 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:12.251238 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"dd7900e01a690d025bb91ee13d4231580c57fc2f4b51e54599ef53b6089de5b7"} Sep 09 17:17:12.251824 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:12.251799 2580 scope.go:117] "RemoveContainer" containerID="88bdfbb030f2182f6cba7aefa49c0ea5c7a2788dcc62f94d3167cd637c7f8776" Sep 09 17:17:12.493508 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:12.493288 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:17:12.493726 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:12.493580 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 09 17:17:12.493937 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:12.493909 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vzr46" Sep 09 17:17:13.124753 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.124724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:13.124906 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.124762 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:13.124906 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.124724 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:13.124906 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:13.124834 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:13.125034 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:13.124940 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:13.125034 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:13.125023 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:13.257842 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.257814 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:17:13.258359 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.258296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" event={"ID":"46c0b0f4-1521-4894-a946-d3c93fb83c39","Type":"ContainerStarted","Data":"86889d62bfba6766ccbe85763709fd5d3a46e0f4e488fa0bc749e6490387b7f7"} Sep 09 17:17:13.258767 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.258544 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:17:13.258767 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.258629 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:17:13.258767 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.258688 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:17:13.275551 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.275526 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:17:13.276321 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.276300 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:17:13.285225 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.285175 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" podStartSLOduration=8.442384847 podStartE2EDuration="27.285157941s" totalImagesPullingTime="18.842773094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:13.283392948 +0000 UTC m=+27.631754421" watchObservedRunningTime="2026-09-09 17:17:13.285157941 +0000 UTC m=+27.633519414" Sep 09 17:17:13.625020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.624979 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xf8st"] Sep 09 17:17:13.625202 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.625147 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:13.625269 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:13.625242 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:13.628358 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.628306 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kkvpl"] Sep 09 17:17:13.628549 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.628431 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:13.628549 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:13.628534 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:13.629116 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.629089 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mt749"] Sep 09 17:17:13.629216 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:13.629188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:13.629298 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:13.629276 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:15.124548 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:15.124521 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:15.125146 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:15.124529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:15.125146 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:15.124621 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:15.125146 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:15.124702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:15.125146 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:15.124532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:15.125146 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:15.124804 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:15.263771 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:15.263741 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bf0bee3-0ee6-4c63-b165-87ddd060ec71" containerID="99bcfc6818978973e62603976b0ee0edcf0937cae0a07a0f2f9018516ada8a8d" exitCode=0 Sep 09 17:17:15.263920 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:15.263820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" event={"ID":"9bf0bee3-0ee6-4c63-b165-87ddd060ec71","Type":"ContainerDied","Data":"99bcfc6818978973e62603976b0ee0edcf0937cae0a07a0f2f9018516ada8a8d"} Sep 09 17:17:17.124672 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:17.124502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:17.125170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:17.124534 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:17.125170 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:17.124757 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:17.125170 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:17.124855 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:17.125170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:17.124536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:17.125170 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:17.124965 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:17.270198 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:17.270170 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bf0bee3-0ee6-4c63-b165-87ddd060ec71" containerID="d7fa8a7d27a59b60d6adebc002e33831a15c0a9543182a2a334d6f5500349882" exitCode=0 Sep 09 17:17:17.270363 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:17.270216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" event={"ID":"9bf0bee3-0ee6-4c63-b165-87ddd060ec71","Type":"ContainerDied","Data":"d7fa8a7d27a59b60d6adebc002e33831a15c0a9543182a2a334d6f5500349882"} Sep 09 17:17:18.821970 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:18.821936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:18.822323 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:18.822040 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 17:17:18.822323 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:18.822092 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret podName:fb607612-0bb9-4046-9e6f-c499d4eb1564 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:50.822078312 +0000 UTC m=+65.170439763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret") pod "global-pull-secret-syncer-mt749" (UID: "fb607612-0bb9-4046-9e6f-c499d4eb1564") : object "kube-system"/"original-pull-secret" not registered Sep 09 17:17:18.922608 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:18.922572 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:18.922790 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:18.922684 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:17:18.922790 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:18.922752 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:50.922737623 +0000 UTC m=+65.271099074 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 17:17:19.023574 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.023537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:19.023764 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:19.023734 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 17:17:19.023764 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:19.023764 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 17:17:19.023893 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:19.023777 2580 projected.go:196] Error preparing data for projected volume kube-api-access-5zq9k for pod openshift-network-diagnostics/network-check-target-xf8st: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:17:19.023893 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:19.023842 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k podName:586fc5a3-1e20-4b14-8380-86ca5ecb6e10 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:51.023823566 +0000 UTC m=+65.372185037 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5zq9k" (UniqueName: "kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k") pod "network-check-target-xf8st" (UID: "586fc5a3-1e20-4b14-8380-86ca5ecb6e10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 17:17:19.124560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.124530 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:19.124658 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.124530 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:19.124658 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:19.124643 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xf8st" podUID="586fc5a3-1e20-4b14-8380-86ca5ecb6e10" Sep 09 17:17:19.124771 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:19.124748 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mt749" podUID="fb607612-0bb9-4046-9e6f-c499d4eb1564" Sep 09 17:17:19.124810 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.124531 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:19.124883 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:19.124867 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:17:19.275843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.275810 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bf0bee3-0ee6-4c63-b165-87ddd060ec71" containerID="dc993303539383bd0b8e45e55ef25e722ac47644ea79cc844ce093306e8b8b4e" exitCode=0 Sep 09 17:17:19.275843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.275852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" event={"ID":"9bf0bee3-0ee6-4c63-b165-87ddd060ec71","Type":"ContainerDied","Data":"dc993303539383bd0b8e45e55ef25e722ac47644ea79cc844ce093306e8b8b4e"} Sep 09 17:17:19.951232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.951163 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-235.ec2.internal" event="NodeReady" Sep 09 17:17:19.951556 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.951264 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 17:17:19.985024 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:19.984988 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-565b7696cc-fptf6"] Sep 09 17:17:20.002115 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.002069 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8j8h9"] Sep 09 17:17:20.002262 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.002135 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.004901 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.004881 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 17:17:20.005051 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.005007 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdgth\"" Sep 09 17:17:20.005131 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.005114 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 17:17:20.005492 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.005473 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 17:17:20.009743 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.009627 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 17:17:20.026033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.026006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565b7696cc-fptf6"] Sep 09 17:17:20.026033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.026037 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c2nh4"] Sep 09 17:17:20.026236 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.026117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.028553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.028530 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 17:17:20.028670 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.028530 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 17:17:20.028864 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.028844 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cblwm\"" Sep 09 17:17:20.050096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.050074 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8j8h9"] Sep 09 17:17:20.050203 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.050106 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c2nh4"] Sep 09 17:17:20.050248 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.050196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c2nh4" Sep 09 17:17:20.052760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.052741 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 17:17:20.052859 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.052823 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 17:17:20.052912 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.052877 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 17:17:20.052975 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.052958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vg7zn\"" Sep 09 17:17:20.132680 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.132857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-config-volume\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.132857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-image-registry-private-configuration\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.132857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-bound-sa-token\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.132857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/daee458d-204d-4371-898f-5094dfe79c09-ca-trust-extracted\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.133068 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvrb\" (UniqueName: \"kubernetes.io/projected/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-kube-api-access-jcvrb\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.133068 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.133068 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-trusted-ca\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.133068 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-installation-pull-secrets\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.133068 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.132985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-tmp-dir\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.133068 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.133014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b76d9\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-kube-api-access-b76d9\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.133340 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.133080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-registry-certificates\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.171091 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.171056 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vzr46_e40de956-6de3-45a1-9117-bd61ee8a6bc1/konnectivity-agent/0.log" limit=1024 Sep 09 17:17:20.233798 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.233720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.233798 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.233758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-config-volume\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.233798 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.233777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-image-registry-private-configuration\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.233812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-bound-sa-token\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.233840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/daee458d-204d-4371-898f-5094dfe79c09-ca-trust-extracted\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234061 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.233861 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:17:20.234061 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.233931 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:17:20.733911585 +0000 UTC m=+35.082273049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:17:20.234061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.233864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvrb\" (UniqueName: \"kubernetes.io/projected/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-kube-api-access-jcvrb\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234085 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-trusted-ca\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-installation-pull-secrets\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-tmp-dir\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.234214 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0376df71-4ec7-4127-94d1-42318c23a135-cert\") pod \"ingress-canary-c2nh4\" (UID: \"0376df71-4ec7-4127-94d1-42318c23a135\") " pod="openshift-ingress-canary/ingress-canary-c2nh4" Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.234229 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b76d9\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-kube-api-access-b76d9\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234323 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.234275 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:20.734261364 +0000 UTC m=+35.082622834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:17:20.234825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234328 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6j7t\" (UniqueName: \"kubernetes.io/projected/0376df71-4ec7-4127-94d1-42318c23a135-kube-api-access-n6j7t\") pod \"ingress-canary-c2nh4\" (UID: \"0376df71-4ec7-4127-94d1-42318c23a135\") " pod="openshift-ingress-canary/ingress-canary-c2nh4" Sep 09 17:17:20.234825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/daee458d-204d-4371-898f-5094dfe79c09-ca-trust-extracted\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234361 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-registry-certificates\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.234825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-config-volume\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.234825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234525 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-tmp-dir\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.235079 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.234922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-registry-certificates\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.235310 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.235288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-trusted-ca\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.238258 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.238234 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-installation-pull-secrets\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.238346 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.238280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-image-registry-private-configuration\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.243740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.243700 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvrb\" (UniqueName: \"kubernetes.io/projected/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-kube-api-access-jcvrb\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.244498 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.244476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b76d9\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-kube-api-access-b76d9\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.246309 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.246287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-bound-sa-token\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.270213 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.270186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vzr46_e40de956-6de3-45a1-9117-bd61ee8a6bc1/konnectivity-agent/0.log" limit=1024 Sep 09 17:17:20.335652 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.335620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0376df71-4ec7-4127-94d1-42318c23a135-cert\") pod \"ingress-canary-c2nh4\" (UID: \"0376df71-4ec7-4127-94d1-42318c23a135\") " pod="openshift-ingress-canary/ingress-canary-c2nh4" Sep 09 17:17:20.335838 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.335685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6j7t\" (UniqueName: \"kubernetes.io/projected/0376df71-4ec7-4127-94d1-42318c23a135-kube-api-access-n6j7t\") pod \"ingress-canary-c2nh4\" (UID: \"0376df71-4ec7-4127-94d1-42318c23a135\") " pod="openshift-ingress-canary/ingress-canary-c2nh4" Sep 09 17:17:20.338614 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.338591 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0376df71-4ec7-4127-94d1-42318c23a135-cert\") pod \"ingress-canary-c2nh4\" (UID: \"0376df71-4ec7-4127-94d1-42318c23a135\") " pod="openshift-ingress-canary/ingress-canary-c2nh4" Sep 09 17:17:20.344160 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.344139 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6j7t\" (UniqueName: \"kubernetes.io/projected/0376df71-4ec7-4127-94d1-42318c23a135-kube-api-access-n6j7t\") pod \"ingress-canary-c2nh4\" (UID: \"0376df71-4ec7-4127-94d1-42318c23a135\") " pod="openshift-ingress-canary/ingress-canary-c2nh4" Sep 09 17:17:20.359015 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.358995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c2nh4" Sep 09 17:17:20.688400 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.688152 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c2nh4"] Sep 09 17:17:20.692377 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:17:20.692354 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0376df71_4ec7_4127_94d1_42318c23a135.slice/crio-5d19ec53321bedad8a491e0c7a2ce03bdcf355bb70e057c56d63c5b6c252a0e8 WatchSource:0}: Error finding container 5d19ec53321bedad8a491e0c7a2ce03bdcf355bb70e057c56d63c5b6c252a0e8: Status 404 returned error can't find the container with id 5d19ec53321bedad8a491e0c7a2ce03bdcf355bb70e057c56d63c5b6c252a0e8 Sep 09 17:17:20.739132 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.739104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:20.739259 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:20.739152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:20.739259 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.739243 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:17:20.739259 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.739254 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:17:20.739368 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.739242 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:17:20.739368 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.739303 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:21.739290076 +0000 UTC m=+36.087651548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:17:20.739368 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:20.739329 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:17:21.739313233 +0000 UTC m=+36.087674684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:17:21.124882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.124847 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:21.125350 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.124847 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:21.125350 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.124975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:21.128020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.127947 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 17:17:21.128160 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.128092 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 17:17:21.128160 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.128118 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 17:17:21.128280 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.128176 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qln5f\"" Sep 09 17:17:21.128406 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.128357 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 17:17:21.128518 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.128504 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kwqx\"" Sep 09 17:17:21.280526 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.280490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c2nh4" event={"ID":"0376df71-4ec7-4127-94d1-42318c23a135","Type":"ContainerStarted","Data":"5d19ec53321bedad8a491e0c7a2ce03bdcf355bb70e057c56d63c5b6c252a0e8"} Sep 09 17:17:21.748445 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.748412 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:21.748737 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:21.748478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:21.748737 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:21.748564 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:17:21.748737 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:21.748578 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:17:21.748737 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:21.748597 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:17:21.748737 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:21.748608 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:17:23.74859537 +0000 UTC m=+38.096956826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:17:21.748737 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:21.748648 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:23.748632194 +0000 UTC m=+38.096993661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:17:23.765316 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:23.765273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:23.765855 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:23.765381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:23.765855 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:23.765468 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:17:23.765855 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:23.765495 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:17:23.765855 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:23.765523 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:17:23.765855 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:23.765562 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:27.765540488 +0000 UTC m=+42.113901939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:17:23.765855 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:23.765588 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:17:27.765569161 +0000 UTC m=+42.113930618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:17:25.122209 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:25.122178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-vzr46_e40de956-6de3-45a1-9117-bd61ee8a6bc1/konnectivity-agent/0.log" limit=1024 Sep 09 17:17:26.293556 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:26.293518 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c2nh4" event={"ID":"0376df71-4ec7-4127-94d1-42318c23a135","Type":"ContainerStarted","Data":"f9030c256436605bac2ddb05eb396caac05585355ec25411c43d53d0a398121f"} Sep 09 17:17:26.296232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:26.296202 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bf0bee3-0ee6-4c63-b165-87ddd060ec71" containerID="a544cf579ce95d86f608c2df2dfd3698cd6636d872c09906e3409037f6ee48a7" exitCode=0 Sep 09 17:17:26.296378 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:26.296252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" event={"ID":"9bf0bee3-0ee6-4c63-b165-87ddd060ec71","Type":"ContainerDied","Data":"a544cf579ce95d86f608c2df2dfd3698cd6636d872c09906e3409037f6ee48a7"} Sep 09 17:17:26.309654 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:26.309613 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c2nh4" podStartSLOduration=2.342068651 podStartE2EDuration="7.309599305s" totalImagesPullingTime="4.967530654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:26.308773491 +0000 UTC m=+40.657134967" watchObservedRunningTime="2026-09-09 17:17:26.309599305 +0000 UTC m=+40.657960778" Sep 09 17:17:27.301595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:27.301562 2580 generic.go:356] "Generic (PLEG): container finished" podID="9bf0bee3-0ee6-4c63-b165-87ddd060ec71" containerID="fec0e2f232419d32ed0ba2f0bde729f25563aa3c34f68218dbb6206f3bd55ae5" exitCode=0 Sep 09 17:17:27.301978 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:27.301649 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" event={"ID":"9bf0bee3-0ee6-4c63-b165-87ddd060ec71","Type":"ContainerDied","Data":"fec0e2f232419d32ed0ba2f0bde729f25563aa3c34f68218dbb6206f3bd55ae5"} Sep 09 17:17:27.794101 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:27.794063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:27.794290 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:27.794227 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:17:27.794290 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:27.794234 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:27.794290 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:27.794249 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:17:27.794449 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:27.794305 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:35.794287175 +0000 UTC m=+50.142648644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:17:27.794449 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:27.794340 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:17:27.794449 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:27.794382 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:17:35.794371304 +0000 UTC m=+50.142732760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:17:28.306865 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:28.306832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" event={"ID":"9bf0bee3-0ee6-4c63-b165-87ddd060ec71","Type":"ContainerStarted","Data":"2cac871b601337aa3551b9f63d41cbb4b5b7477c324db669c69589f2acccbe14"} Sep 09 17:17:28.327932 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:28.327893 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nr8h7" podStartSLOduration=7.525397765 podStartE2EDuration="42.327878688s" totalImagesPullingTime="34.802480923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:28.326148384 +0000 UTC m=+42.674509878" watchObservedRunningTime="2026-09-09 17:17:28.327878688 +0000 UTC m=+42.676240176" Sep 09 17:17:35.845632 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:35.845576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:35.846065 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:35.845729 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:17:35.846065 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:35.845741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:35.846065 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:35.845748 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:17:35.846065 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:35.845796 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:17:51.845782154 +0000 UTC m=+66.194143606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:17:35.846065 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:35.845846 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:17:35.846065 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:35.845889 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:17:51.8458784 +0000 UTC m=+66.194239856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:17:45.274322 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:45.274293 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z4nmc" Sep 09 17:17:50.856041 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:50.855987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:50.858780 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:50.858762 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 17:17:50.869844 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:50.869812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fb607612-0bb9-4046-9e6f-c499d4eb1564-original-pull-secret\") pod \"global-pull-secret-syncer-mt749\" (UID: \"fb607612-0bb9-4046-9e6f-c499d4eb1564\") " pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:50.956526 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:50.956492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:17:50.959147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:50.959126 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 17:17:50.967328 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:50.967306 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 17:17:50.967407 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:50.967374 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:54.967358178 +0000 UTC m=+129.315719629 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : secret "metrics-daemon-secret" not found Sep 09 17:17:51.057423 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.057376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:51.060302 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.060284 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 17:17:51.070008 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.069988 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 17:17:51.080790 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.080766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zq9k\" (UniqueName: \"kubernetes.io/projected/586fc5a3-1e20-4b14-8380-86ca5ecb6e10-kube-api-access-5zq9k\") pod \"network-check-target-xf8st\" (UID: \"586fc5a3-1e20-4b14-8380-86ca5ecb6e10\") " pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:51.140667 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.140587 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qln5f\"" Sep 09 17:17:51.144788 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.144770 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mt749" Sep 09 17:17:51.148535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.148519 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:51.423706 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.423629 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mt749"] Sep 09 17:17:51.426221 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.426194 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xf8st"] Sep 09 17:17:51.427248 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:17:51.427219 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb607612_0bb9_4046_9e6f_c499d4eb1564.slice/crio-b351a7a8b5f03e281c15c9e6a55cedb3b66607d9d4be5249c809049ef2faad5b WatchSource:0}: Error finding container b351a7a8b5f03e281c15c9e6a55cedb3b66607d9d4be5249c809049ef2faad5b: Status 404 returned error can't find the container with id b351a7a8b5f03e281c15c9e6a55cedb3b66607d9d4be5249c809049ef2faad5b Sep 09 17:17:51.429814 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:17:51.429789 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586fc5a3_1e20_4b14_8380_86ca5ecb6e10.slice/crio-9aa53150f8bf62d7069c42a75e51cde3f89b7f26bc2f60b33124b56af3337949 WatchSource:0}: Error finding container 9aa53150f8bf62d7069c42a75e51cde3f89b7f26bc2f60b33124b56af3337949: Status 404 returned error can't find the container with id 9aa53150f8bf62d7069c42a75e51cde3f89b7f26bc2f60b33124b56af3337949 Sep 09 17:17:51.863462 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.863428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:17:51.863900 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:51.863479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:17:51.863900 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:51.863582 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:17:51.863900 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:51.863593 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:17:51.863900 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:51.863625 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:17:51.863900 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:51.863642 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:18:23.863629207 +0000 UTC m=+98.211990658 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:17:51.863900 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:17:51.863693 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:18:23.863672789 +0000 UTC m=+98.212034254 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:17:52.366191 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:52.366144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xf8st" event={"ID":"586fc5a3-1e20-4b14-8380-86ca5ecb6e10","Type":"ContainerStarted","Data":"9aa53150f8bf62d7069c42a75e51cde3f89b7f26bc2f60b33124b56af3337949"} Sep 09 17:17:52.367464 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:52.367411 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mt749" event={"ID":"fb607612-0bb9-4046-9e6f-c499d4eb1564","Type":"ContainerStarted","Data":"b351a7a8b5f03e281c15c9e6a55cedb3b66607d9d4be5249c809049ef2faad5b"} Sep 09 17:17:57.381794 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:57.381748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xf8st" event={"ID":"586fc5a3-1e20-4b14-8380-86ca5ecb6e10","Type":"ContainerStarted","Data":"f77c8b329cc26bb1dbf2ca74b35c87058a3887d8a139c34a5ca37fd31714b37c"} Sep 09 17:17:57.382271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:57.381823 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:17:57.386291 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:57.386257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mt749" event={"ID":"fb607612-0bb9-4046-9e6f-c499d4eb1564","Type":"ContainerStarted","Data":"4d0c80f7bb0c1ff42d5a4e793c7d310ccd7cba6043a85db6a8cbedcd7cf5e903"} Sep 09 17:17:57.397074 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:57.397022 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xf8st" podStartSLOduration=66.210848575 podStartE2EDuration="1m11.397008136s" totalImagesPullingTime="5.186159561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:57.396766635 +0000 UTC m=+71.745128104" watchObservedRunningTime="2026-09-09 17:17:57.397008136 +0000 UTC m=+71.745369600" Sep 09 17:17:57.411296 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:17:57.411232 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mt749" podStartSLOduration=66.217861344 podStartE2EDuration="1m11.411212781s" totalImagesPullingTime="5.193351437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:17:57.410458781 +0000 UTC m=+71.758820254" watchObservedRunningTime="2026-09-09 17:17:57.411212781 +0000 UTC m=+71.759574254" Sep 09 17:18:23.896404 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:18:23.896371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:18:23.896853 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:18:23.896427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:18:23.896853 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:18:23.896535 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:18:23.896853 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:18:23.896556 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:18:23.896853 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:18:23.896585 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:18:23.896853 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:18:23.896630 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:19:27.896613537 +0000 UTC m=+162.244974991 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:18:23.896853 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:18:23.896644 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:19:27.896637926 +0000 UTC m=+162.244999377 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:18:28.390515 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:18:28.390485 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xf8st" Sep 09 17:18:55.002548 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:18:55.002505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:18:55.002990 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:18:55.002627 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 17:18:55.002990 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:18:55.002687 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:20:57.002674898 +0000 UTC m=+251.351036350 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : secret "metrics-daemon-secret" not found Sep 09 17:19:23.012427 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:19:23.012390 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" podUID="daee458d-204d-4371-898f-5094dfe79c09" Sep 09 17:19:23.036759 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:19:23.036721 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8j8h9" podUID="b79ceb92-b128-4b3b-9fa5-64ae8bbb384f" Sep 09 17:19:23.591175 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:19:23.591148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:19:23.591327 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:19:23.591148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8j8h9" Sep 09 17:19:24.151167 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:19:24.151134 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:19:27.915621 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:19:27.915566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:19:27.916049 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:19:27.915653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:19:27.916049 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:19:27.915749 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 17:19:27.916049 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:19:27.915764 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 17:19:27.916049 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:19:27.915766 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-565b7696cc-fptf6: secret "image-registry-tls" not found Sep 09 17:19:27.916049 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:19:27.915841 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls podName:b79ceb92-b128-4b3b-9fa5-64ae8bbb384f nodeName:}" failed. No retries permitted until 2026-09-09 17:21:29.915827925 +0000 UTC m=+284.264189380 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls") pod "dns-default-8j8h9" (UID: "b79ceb92-b128-4b3b-9fa5-64ae8bbb384f") : secret "dns-default-metrics-tls" not found Sep 09 17:19:27.916049 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:19:27.915855 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls podName:daee458d-204d-4371-898f-5094dfe79c09 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:29.915850021 +0000 UTC m=+284.264211472 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls") pod "image-registry-565b7696cc-fptf6" (UID: "daee458d-204d-4371-898f-5094dfe79c09") : secret "image-registry-tls" not found Sep 09 17:19:36.125821 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:19:36.125793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:20:52.987262 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.987226 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z9596"] Sep 09 17:20:52.989947 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.989928 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz"] Sep 09 17:20:52.990076 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.990054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:52.992406 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.992389 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5"] Sep 09 17:20:52.992521 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.992505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:52.993384 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.993367 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 09 17:20:52.993474 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.993390 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:20:52.993474 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.993415 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 09 17:20:52.994479 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.994464 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 09 17:20:52.994677 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.994665 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-94ws4\"" Sep 09 17:20:52.994905 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.994886 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 09 17:20:52.995189 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.995161 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:52.996278 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.996259 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 17:20:52.996485 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.996321 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 17:20:52.997658 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.997640 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 09 17:20:52.997767 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.997637 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5xzsn\"" Sep 09 17:20:52.998350 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.998330 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 09 17:20:52.998579 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.998565 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 09 17:20:52.998815 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.998796 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 09 17:20:52.998907 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.998803 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hlswd\"" Sep 09 17:20:52.998970 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:52.998957 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:20:53.000337 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.000320 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 09 17:20:53.004581 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.004553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z9596"] Sep 09 17:20:53.006159 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.006137 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5"] Sep 09 17:20:53.011169 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.011138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz"] Sep 09 17:20:53.077222 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.077187 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-rxfld"] Sep 09 17:20:53.080132 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.080108 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-rxfld" Sep 09 17:20:53.081087 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.081067 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h"] Sep 09 17:20:53.082565 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.082550 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-dv496\"" Sep 09 17:20:53.083690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.083672 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x"] Sep 09 17:20:53.083805 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.083791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.086488 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.086468 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:53.086661 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.086641 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 09 17:20:53.086828 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.086807 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 09 17:20:53.087085 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.087068 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 09 17:20:53.087157 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.087131 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:20:53.087431 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.087415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9hb5z\"" Sep 09 17:20:53.088805 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.088787 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 09 17:20:53.088904 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.088873 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 09 17:20:53.089166 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.089151 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:20:53.089705 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.089684 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nll8x\"" Sep 09 17:20:53.091243 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.091226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-rxfld"] Sep 09 17:20:53.097729 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.095844 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h"] Sep 09 17:20:53.099927 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.099906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x"] Sep 09 17:20:53.181434 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/930aa645-7318-4c90-850d-560c1430ab0f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.181598 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdt4k\" (UniqueName: \"kubernetes.io/projected/fc4c84c4-17b1-44f5-92d7-c8cec5aaab7e-kube-api-access-pdt4k\") pod \"network-check-source-685456655f-rxfld\" (UID: \"fc4c84c4-17b1-44f5-92d7-c8cec5aaab7e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-rxfld" Sep 09 17:20:53.181598 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bwcx\" (UniqueName: \"kubernetes.io/projected/930aa645-7318-4c90-850d-560c1430ab0f-kube-api-access-7bwcx\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.181598 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181521 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d11da5db-6654-463b-8880-b7aade5a657f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.181746 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181600 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4e0a01-e009-4a59-9826-b82707527015-serving-cert\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.181746 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b4e0a01-e009-4a59-9826-b82707527015-trusted-ca\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.181746 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpr2m\" (UniqueName: \"kubernetes.io/projected/e8028556-104d-4b1d-9914-138bf2e3f97b-kube-api-access-cpr2m\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.181746 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcxzr\" (UniqueName: \"kubernetes.io/projected/810557d8-1bd0-4e5f-a5f8-af18803cb208-kube-api-access-kcxzr\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:53.181746 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8028556-104d-4b1d-9914-138bf2e3f97b-config\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.181932 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181798 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49xrc\" (UniqueName: \"kubernetes.io/projected/9b4e0a01-e009-4a59-9826-b82707527015-kube-api-access-49xrc\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.181932 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.181932 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d11da5db-6654-463b-8880-b7aade5a657f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.181932 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8028556-104d-4b1d-9914-138bf2e3f97b-serving-cert\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.181932 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b4e0a01-e009-4a59-9826-b82707527015-config\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.182081 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcj7l\" (UniqueName: \"kubernetes.io/projected/d11da5db-6654-463b-8880-b7aade5a657f-kube-api-access-bcj7l\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.182081 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.181955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:53.282505 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpr2m\" (UniqueName: \"kubernetes.io/projected/e8028556-104d-4b1d-9914-138bf2e3f97b-kube-api-access-cpr2m\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.282505 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcxzr\" (UniqueName: \"kubernetes.io/projected/810557d8-1bd0-4e5f-a5f8-af18803cb208-kube-api-access-kcxzr\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8028556-104d-4b1d-9914-138bf2e3f97b-config\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49xrc\" (UniqueName: \"kubernetes.io/projected/9b4e0a01-e009-4a59-9826-b82707527015-kube-api-access-49xrc\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d11da5db-6654-463b-8880-b7aade5a657f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8028556-104d-4b1d-9914-138bf2e3f97b-serving-cert\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b4e0a01-e009-4a59-9826-b82707527015-config\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:53.282681 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcj7l\" (UniqueName: \"kubernetes.io/projected/d11da5db-6654-463b-8880-b7aade5a657f-kube-api-access-bcj7l\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.282757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:53.282774 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls podName:930aa645-7318-4c90-850d-560c1430ab0f nodeName:}" failed. No retries permitted until 2026-09-09 17:20:53.782754635 +0000 UTC m=+248.131116100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nh2hz" (UID: "930aa645-7318-4c90-850d-560c1430ab0f") : secret "cluster-monitoring-operator-tls" not found Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/930aa645-7318-4c90-850d-560c1430ab0f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:53.282819 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdt4k\" (UniqueName: \"kubernetes.io/projected/fc4c84c4-17b1-44f5-92d7-c8cec5aaab7e-kube-api-access-pdt4k\") pod \"network-check-source-685456655f-rxfld\" (UID: \"fc4c84c4-17b1-44f5-92d7-c8cec5aaab7e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-rxfld" Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:53.282879 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls podName:810557d8-1bd0-4e5f-a5f8-af18803cb208 nodeName:}" failed. No retries permitted until 2026-09-09 17:20:53.782862704 +0000 UTC m=+248.131224166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-fks5x" (UID: "810557d8-1bd0-4e5f-a5f8-af18803cb208") : secret "samples-operator-tls" not found Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bwcx\" (UniqueName: \"kubernetes.io/projected/930aa645-7318-4c90-850d-560c1430ab0f-kube-api-access-7bwcx\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d11da5db-6654-463b-8880-b7aade5a657f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.282981 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4e0a01-e009-4a59-9826-b82707527015-serving-cert\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.283005 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b4e0a01-e009-4a59-9826-b82707527015-trusted-ca\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.283197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.283144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8028556-104d-4b1d-9914-138bf2e3f97b-config\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.283692 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.283460 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d11da5db-6654-463b-8880-b7aade5a657f-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.283692 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.283570 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b4e0a01-e009-4a59-9826-b82707527015-config\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.283897 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.283875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/930aa645-7318-4c90-850d-560c1430ab0f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.284024 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.284001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b4e0a01-e009-4a59-9826-b82707527015-trusted-ca\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.285303 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.285277 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b4e0a01-e009-4a59-9826-b82707527015-serving-cert\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.285394 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.285365 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d11da5db-6654-463b-8880-b7aade5a657f-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.285446 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.285429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8028556-104d-4b1d-9914-138bf2e3f97b-serving-cert\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.291613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.291575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdt4k\" (UniqueName: \"kubernetes.io/projected/fc4c84c4-17b1-44f5-92d7-c8cec5aaab7e-kube-api-access-pdt4k\") pod \"network-check-source-685456655f-rxfld\" (UID: \"fc4c84c4-17b1-44f5-92d7-c8cec5aaab7e\") " pod="openshift-network-diagnostics/network-check-source-685456655f-rxfld" Sep 09 17:20:53.291849 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.291824 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bwcx\" (UniqueName: \"kubernetes.io/projected/930aa645-7318-4c90-850d-560c1430ab0f-kube-api-access-7bwcx\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.292040 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.292021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcxzr\" (UniqueName: \"kubernetes.io/projected/810557d8-1bd0-4e5f-a5f8-af18803cb208-kube-api-access-kcxzr\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:53.292271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.292254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcj7l\" (UniqueName: \"kubernetes.io/projected/d11da5db-6654-463b-8880-b7aade5a657f-kube-api-access-bcj7l\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8sl5\" (UID: \"d11da5db-6654-463b-8880-b7aade5a657f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.292305 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.292271 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49xrc\" (UniqueName: \"kubernetes.io/projected/9b4e0a01-e009-4a59-9826-b82707527015-kube-api-access-49xrc\") pod \"console-operator-85b454874b-z9596\" (UID: \"9b4e0a01-e009-4a59-9826-b82707527015\") " pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.292945 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.292928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpr2m\" (UniqueName: \"kubernetes.io/projected/e8028556-104d-4b1d-9914-138bf2e3f97b-kube-api-access-cpr2m\") pod \"service-ca-operator-859f94c4b-4kj5h\" (UID: \"e8028556-104d-4b1d-9914-138bf2e3f97b\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.299870 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.299852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:20:53.311774 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.311758 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" Sep 09 17:20:53.391148 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.391119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-rxfld" Sep 09 17:20:53.401257 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.401239 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" Sep 09 17:20:53.566653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.566574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-z9596"] Sep 09 17:20:53.569372 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:20:53.569334 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4e0a01_e009_4a59_9826_b82707527015.slice/crio-b9170fd278ac5b5cc3a5b97fe8d17341c6e94e305a7d8a23e29d8fe76db93ac8 WatchSource:0}: Error finding container b9170fd278ac5b5cc3a5b97fe8d17341c6e94e305a7d8a23e29d8fe76db93ac8: Status 404 returned error can't find the container with id b9170fd278ac5b5cc3a5b97fe8d17341c6e94e305a7d8a23e29d8fe76db93ac8 Sep 09 17:20:53.582195 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.582168 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5"] Sep 09 17:20:53.584561 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:20:53.584543 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11da5db_6654_463b_8880_b7aade5a657f.slice/crio-2c7a1b807871c937e715b314d5132cd1409c432dc4a76f9105ebb685edc4936c WatchSource:0}: Error finding container 2c7a1b807871c937e715b314d5132cd1409c432dc4a76f9105ebb685edc4936c: Status 404 returned error can't find the container with id 2c7a1b807871c937e715b314d5132cd1409c432dc4a76f9105ebb685edc4936c Sep 09 17:20:53.658365 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.658334 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-rxfld"] Sep 09 17:20:53.661074 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:20:53.661052 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4c84c4_17b1_44f5_92d7_c8cec5aaab7e.slice/crio-4337071f857843541af53bdae5a360972983dd3af9b37d90d27e05da84891d8c WatchSource:0}: Error finding container 4337071f857843541af53bdae5a360972983dd3af9b37d90d27e05da84891d8c: Status 404 returned error can't find the container with id 4337071f857843541af53bdae5a360972983dd3af9b37d90d27e05da84891d8c Sep 09 17:20:53.682482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.682379 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h"] Sep 09 17:20:53.686596 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:20:53.686574 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8028556_104d_4b1d_9914_138bf2e3f97b.slice/crio-e346286065e9b745092001abf02d8766b29281285fab933783faa03e36bae573 WatchSource:0}: Error finding container e346286065e9b745092001abf02d8766b29281285fab933783faa03e36bae573: Status 404 returned error can't find the container with id e346286065e9b745092001abf02d8766b29281285fab933783faa03e36bae573 Sep 09 17:20:53.787260 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.787223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:53.787416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.787312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:53.787416 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:53.787385 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 17:20:53.787488 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:53.787444 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 17:20:53.787488 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:53.787454 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls podName:930aa645-7318-4c90-850d-560c1430ab0f nodeName:}" failed. No retries permitted until 2026-09-09 17:20:54.787436738 +0000 UTC m=+249.135798197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nh2hz" (UID: "930aa645-7318-4c90-850d-560c1430ab0f") : secret "cluster-monitoring-operator-tls" not found Sep 09 17:20:53.787570 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:53.787493 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls podName:810557d8-1bd0-4e5f-a5f8-af18803cb208 nodeName:}" failed. No retries permitted until 2026-09-09 17:20:54.787478366 +0000 UTC m=+249.135839823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-fks5x" (UID: "810557d8-1bd0-4e5f-a5f8-af18803cb208") : secret "samples-operator-tls" not found Sep 09 17:20:53.799367 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.799336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" event={"ID":"e8028556-104d-4b1d-9914-138bf2e3f97b","Type":"ContainerStarted","Data":"e346286065e9b745092001abf02d8766b29281285fab933783faa03e36bae573"} Sep 09 17:20:53.800456 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.800434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" event={"ID":"d11da5db-6654-463b-8880-b7aade5a657f","Type":"ContainerStarted","Data":"2c7a1b807871c937e715b314d5132cd1409c432dc4a76f9105ebb685edc4936c"} Sep 09 17:20:53.801521 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.801499 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z9596" event={"ID":"9b4e0a01-e009-4a59-9826-b82707527015","Type":"ContainerStarted","Data":"b9170fd278ac5b5cc3a5b97fe8d17341c6e94e305a7d8a23e29d8fe76db93ac8"} Sep 09 17:20:53.802937 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.802916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-rxfld" event={"ID":"fc4c84c4-17b1-44f5-92d7-c8cec5aaab7e","Type":"ContainerStarted","Data":"1e76098209424a658e58dd84c54e9e9c26f4dcea543372f8a62e01d7dda9b8b3"} Sep 09 17:20:53.803014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.802957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-rxfld" event={"ID":"fc4c84c4-17b1-44f5-92d7-c8cec5aaab7e","Type":"ContainerStarted","Data":"4337071f857843541af53bdae5a360972983dd3af9b37d90d27e05da84891d8c"} Sep 09 17:20:53.817196 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:53.817128 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-rxfld" podStartSLOduration=0.817114963 podStartE2EDuration="817.114963ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:20:53.816453314 +0000 UTC m=+248.164814786" watchObservedRunningTime="2026-09-09 17:20:53.817114963 +0000 UTC m=+248.165476435" Sep 09 17:20:54.796541 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:54.796503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:54.797032 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:54.796602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:54.797032 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:54.796654 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 17:20:54.797032 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:54.796724 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 17:20:54.797032 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:54.796753 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls podName:930aa645-7318-4c90-850d-560c1430ab0f nodeName:}" failed. No retries permitted until 2026-09-09 17:20:56.796732618 +0000 UTC m=+251.145094089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nh2hz" (UID: "930aa645-7318-4c90-850d-560c1430ab0f") : secret "cluster-monitoring-operator-tls" not found Sep 09 17:20:54.797032 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:54.796778 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls podName:810557d8-1bd0-4e5f-a5f8-af18803cb208 nodeName:}" failed. No retries permitted until 2026-09-09 17:20:56.79676209 +0000 UTC m=+251.145123548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-fks5x" (UID: "810557d8-1bd0-4e5f-a5f8-af18803cb208") : secret "samples-operator-tls" not found Sep 09 17:20:56.575881 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.575856 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wjqgn"] Sep 09 17:20:56.580402 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.580380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.583780 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.583752 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 17:20:56.583970 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.583929 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 17:20:56.584204 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.584146 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 17:20:56.586758 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.586737 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 17:20:56.586826 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.586796 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wvqv8\"" Sep 09 17:20:56.595721 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.595682 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wjqgn"] Sep 09 17:20:56.711994 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.711971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.712107 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.712007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-data-volume\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.712107 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.712055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-crio-socket\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.712194 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.712132 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.712277 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.712250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zsgf\" (UniqueName: \"kubernetes.io/projected/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-api-access-2zsgf\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.813330 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.813472 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-data-volume\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.813472 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-crio-socket\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.813472 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:56.813404 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:20:56.813472 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:56.813474 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:20:57.313455305 +0000 UTC m=+251.661816768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : configmap "kube-rbac-proxy" not found Sep 09 17:20:56.813690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813530 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-crio-socket\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.813690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:20:56.813690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.813690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zsgf\" (UniqueName: \"kubernetes.io/projected/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-api-access-2zsgf\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.813902 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813687 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-data-volume\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.813902 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.813727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:20:56.813902 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:56.813811 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:20:56.813902 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:56.813827 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 17:20:56.813902 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:56.813837 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 17:20:56.813902 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:56.813861 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:20:57.313845499 +0000 UTC m=+251.662206954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : secret "insights-runtime-extractor-tls" not found Sep 09 17:20:56.813902 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:56.813880 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls podName:930aa645-7318-4c90-850d-560c1430ab0f nodeName:}" failed. No retries permitted until 2026-09-09 17:21:00.813870757 +0000 UTC m=+255.162232215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nh2hz" (UID: "930aa645-7318-4c90-850d-560c1430ab0f") : secret "cluster-monitoring-operator-tls" not found Sep 09 17:20:56.813902 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:56.813894 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls podName:810557d8-1bd0-4e5f-a5f8-af18803cb208 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:00.813886348 +0000 UTC m=+255.162247801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-fks5x" (UID: "810557d8-1bd0-4e5f-a5f8-af18803cb208") : secret "samples-operator-tls" not found Sep 09 17:20:56.814824 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.814796 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" event={"ID":"e8028556-104d-4b1d-9914-138bf2e3f97b","Type":"ContainerStarted","Data":"5158251381c0b2a6b601f30616a3ff988eca806c1a568ee767315ea2ee7bcd34"} Sep 09 17:20:56.818233 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.818207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" event={"ID":"d11da5db-6654-463b-8880-b7aade5a657f","Type":"ContainerStarted","Data":"5a649647c09d474528363d88a2cc57c8363f32915cbb5d0f2c9f50b2e5fa0acd"} Sep 09 17:20:56.820697 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.820678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/0.log" Sep 09 17:20:56.820811 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.820736 2580 generic.go:356] "Generic (PLEG): container finished" podID="9b4e0a01-e009-4a59-9826-b82707527015" containerID="42dda370034f4d48e8dde7db9c4b25f3f1717b9f304607cbe5ff2cce564ef48d" exitCode=255 Sep 09 17:20:56.820871 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.820807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z9596" event={"ID":"9b4e0a01-e009-4a59-9826-b82707527015","Type":"ContainerDied","Data":"42dda370034f4d48e8dde7db9c4b25f3f1717b9f304607cbe5ff2cce564ef48d"} Sep 09 17:20:56.821027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.821002 2580 scope.go:117] "RemoveContainer" containerID="42dda370034f4d48e8dde7db9c4b25f3f1717b9f304607cbe5ff2cce564ef48d" Sep 09 17:20:56.823021 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.822999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zsgf\" (UniqueName: \"kubernetes.io/projected/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-api-access-2zsgf\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:56.831138 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.831092 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" podStartSLOduration=0.958361368 podStartE2EDuration="3.831074432s" totalImagesPullingTime="2.872713064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:20:56.830224336 +0000 UTC m=+251.178585810" watchObservedRunningTime="2026-09-09 17:20:56.831074432 +0000 UTC m=+251.179435905" Sep 09 17:20:56.858291 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:56.858239 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" podStartSLOduration=1.889238 podStartE2EDuration="4.858188507s" totalImagesPullingTime="2.968950507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:20:56.857217539 +0000 UTC m=+251.205579009" watchObservedRunningTime="2026-09-09 17:20:56.858188507 +0000 UTC m=+251.206549980" Sep 09 17:20:57.015977 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.015940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:20:57.016129 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:57.016112 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 17:20:57.016197 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:57.016187 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs podName:f9649a0b-cc7d-4eb0-9952-865e10002bd7 nodeName:}" failed. No retries permitted until 2026-09-09 17:22:59.01616901 +0000 UTC m=+373.364530466 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs") pod "network-metrics-daemon-kkvpl" (UID: "f9649a0b-cc7d-4eb0-9952-865e10002bd7") : secret "metrics-daemon-secret" not found Sep 09 17:20:57.318620 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.318582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:57.318806 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.318656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:57.318806 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:57.318755 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:20:57.318806 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:57.318779 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:20:57.318935 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:57.318817 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:20:58.318802872 +0000 UTC m=+252.667164328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : secret "insights-runtime-extractor-tls" not found Sep 09 17:20:57.318935 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:57.318831 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:20:58.318824942 +0000 UTC m=+252.667186393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : configmap "kube-rbac-proxy" not found Sep 09 17:20:57.825455 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.825370 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:20:57.825904 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.825862 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/0.log" Sep 09 17:20:57.825959 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.825899 2580 generic.go:356] "Generic (PLEG): container finished" podID="9b4e0a01-e009-4a59-9826-b82707527015" containerID="20bc98b7e3038a3367fde8fac97c3aad568484e7d699f15b3bdddf000fc9111a" exitCode=255 Sep 09 17:20:57.825959 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.825937 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z9596" event={"ID":"9b4e0a01-e009-4a59-9826-b82707527015","Type":"ContainerDied","Data":"20bc98b7e3038a3367fde8fac97c3aad568484e7d699f15b3bdddf000fc9111a"} Sep 09 17:20:57.826055 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.825997 2580 scope.go:117] "RemoveContainer" containerID="42dda370034f4d48e8dde7db9c4b25f3f1717b9f304607cbe5ff2cce564ef48d" Sep 09 17:20:57.826271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.826253 2580 scope.go:117] "RemoveContainer" containerID="20bc98b7e3038a3367fde8fac97c3aad568484e7d699f15b3bdddf000fc9111a" Sep 09 17:20:57.826501 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:57.826474 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z9596_openshift-console-operator(9b4e0a01-e009-4a59-9826-b82707527015)\"" pod="openshift-console-operator/console-operator-85b454874b-z9596" podUID="9b4e0a01-e009-4a59-9826-b82707527015" Sep 09 17:20:57.893428 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.893393 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g"] Sep 09 17:20:57.897457 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.897437 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" Sep 09 17:20:57.900087 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.900065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8fnvz\"" Sep 09 17:20:57.900191 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.900102 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 09 17:20:57.900264 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.900241 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 09 17:20:57.904547 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.904527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g"] Sep 09 17:20:57.921995 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:57.921965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtd49\" (UniqueName: \"kubernetes.io/projected/2d3aafe0-62ec-46e3-8d0b-aeb98d80060c-kube-api-access-wtd49\") pod \"migrator-fdbb8fdff-5647g\" (UID: \"2d3aafe0-62ec-46e3-8d0b-aeb98d80060c\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" Sep 09 17:20:58.023433 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.023397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtd49\" (UniqueName: \"kubernetes.io/projected/2d3aafe0-62ec-46e3-8d0b-aeb98d80060c-kube-api-access-wtd49\") pod \"migrator-fdbb8fdff-5647g\" (UID: \"2d3aafe0-62ec-46e3-8d0b-aeb98d80060c\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" Sep 09 17:20:58.031184 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.031155 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtd49\" (UniqueName: \"kubernetes.io/projected/2d3aafe0-62ec-46e3-8d0b-aeb98d80060c-kube-api-access-wtd49\") pod \"migrator-fdbb8fdff-5647g\" (UID: \"2d3aafe0-62ec-46e3-8d0b-aeb98d80060c\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" Sep 09 17:20:58.207664 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.207592 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" Sep 09 17:20:58.325912 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.325878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:58.326055 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.326019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:20:58.326055 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:58.326019 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:20:58.326126 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:58.326091 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:00.326076876 +0000 UTC m=+254.674438331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : secret "insights-runtime-extractor-tls" not found Sep 09 17:20:58.326175 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:58.326129 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:20:58.326209 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:58.326177 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:00.326164932 +0000 UTC m=+254.674526390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : configmap "kube-rbac-proxy" not found Sep 09 17:20:58.335297 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.335274 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bkxw4_07eab57a-6bbe-4919-8121-a5fb27de7f48/dns-node-resolver/0.log" Sep 09 17:20:58.470755 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.470671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g"] Sep 09 17:20:58.473642 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:20:58.473611 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3aafe0_62ec_46e3_8d0b_aeb98d80060c.slice/crio-92e2c3cbcb1977bf72db2844926556fb834136908af9f8ca0bdaf4007119b3d6 WatchSource:0}: Error finding container 92e2c3cbcb1977bf72db2844926556fb834136908af9f8ca0bdaf4007119b3d6: Status 404 returned error can't find the container with id 92e2c3cbcb1977bf72db2844926556fb834136908af9f8ca0bdaf4007119b3d6 Sep 09 17:20:58.834371 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.834341 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:20:58.834831 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.834726 2580 scope.go:117] "RemoveContainer" containerID="20bc98b7e3038a3367fde8fac97c3aad568484e7d699f15b3bdddf000fc9111a" Sep 09 17:20:58.834918 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:20:58.834893 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z9596_openshift-console-operator(9b4e0a01-e009-4a59-9826-b82707527015)\"" pod="openshift-console-operator/console-operator-85b454874b-z9596" podUID="9b4e0a01-e009-4a59-9826-b82707527015" Sep 09 17:20:58.835574 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:58.835554 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" event={"ID":"2d3aafe0-62ec-46e3-8d0b-aeb98d80060c","Type":"ContainerStarted","Data":"92e2c3cbcb1977bf72db2844926556fb834136908af9f8ca0bdaf4007119b3d6"} Sep 09 17:20:59.535503 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:59.535469 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vcxxj_44fac9f1-d2e6-4397-bb60-47cf60343669/node-ca/0.log" Sep 09 17:20:59.840579 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:59.840546 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" event={"ID":"2d3aafe0-62ec-46e3-8d0b-aeb98d80060c","Type":"ContainerStarted","Data":"6c50cb14a2728e7c35d24028ebc4bb569da0f9e6484ff092dc7bc07894800631"} Sep 09 17:20:59.840888 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:59.840609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" event={"ID":"2d3aafe0-62ec-46e3-8d0b-aeb98d80060c","Type":"ContainerStarted","Data":"e292baf0f81d0324fc38eac161d0ae2df4637affb457772566c277d2ad103c20"} Sep 09 17:20:59.854347 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:20:59.854293 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-5647g" podStartSLOduration=1.650324662 podStartE2EDuration="2.854272687s" totalImagesPullingTime="1.203948025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:20:59.85337272 +0000 UTC m=+254.201734200" watchObservedRunningTime="2026-09-09 17:20:59.854272687 +0000 UTC m=+254.202634160" Sep 09 17:21:00.135659 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:00.135555 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2nh4_0376df71-4ec7-4127-94d1-42318c23a135/serve-healthcheck-canary/0.log" Sep 09 17:21:00.344153 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:00.344114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:21:00.344326 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:00.344193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:21:00.344326 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:00.344236 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 17:21:00.344326 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:00.344301 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:04.344286271 +0000 UTC m=+258.692647728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : configmap "kube-rbac-proxy" not found Sep 09 17:21:00.344436 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:00.344329 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:21:00.344436 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:00.344382 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:04.34436994 +0000 UTC m=+258.692731392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : secret "insights-runtime-extractor-tls" not found Sep 09 17:21:00.847100 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:00.847068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:21:00.847535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:00.847129 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:21:00.847535 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:00.847212 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 17:21:00.847535 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:00.847239 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 17:21:00.847535 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:00.847257 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls podName:810557d8-1bd0-4e5f-a5f8-af18803cb208 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:08.847245685 +0000 UTC m=+263.195607136 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-fks5x" (UID: "810557d8-1bd0-4e5f-a5f8-af18803cb208") : secret "samples-operator-tls" not found Sep 09 17:21:00.847535 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:00.847303 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls podName:930aa645-7318-4c90-850d-560c1430ab0f nodeName:}" failed. No retries permitted until 2026-09-09 17:21:08.847285468 +0000 UTC m=+263.195646919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nh2hz" (UID: "930aa645-7318-4c90-850d-560c1430ab0f") : secret "cluster-monitoring-operator-tls" not found Sep 09 17:21:03.300415 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:03.300328 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:21:03.300779 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:03.300410 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:21:03.300779 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:03.300705 2580 scope.go:117] "RemoveContainer" containerID="20bc98b7e3038a3367fde8fac97c3aad568484e7d699f15b3bdddf000fc9111a" Sep 09 17:21:03.300900 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:03.300881 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z9596_openshift-console-operator(9b4e0a01-e009-4a59-9826-b82707527015)\"" pod="openshift-console-operator/console-operator-85b454874b-z9596" podUID="9b4e0a01-e009-4a59-9826-b82707527015" Sep 09 17:21:03.852503 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:03.852474 2580 scope.go:117] "RemoveContainer" containerID="20bc98b7e3038a3367fde8fac97c3aad568484e7d699f15b3bdddf000fc9111a" Sep 09 17:21:03.852668 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:03.852623 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-z9596_openshift-console-operator(9b4e0a01-e009-4a59-9826-b82707527015)\"" pod="openshift-console-operator/console-operator-85b454874b-z9596" podUID="9b4e0a01-e009-4a59-9826-b82707527015" Sep 09 17:21:04.377475 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:04.377422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:21:04.377880 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:04.377510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:21:04.377880 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:04.377588 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 17:21:04.377880 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:04.377655 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls podName:bcd3c8d8-2c37-43ec-8cab-ed05b059bb75 nodeName:}" failed. No retries permitted until 2026-09-09 17:21:12.377635628 +0000 UTC m=+266.725997099 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wjqgn" (UID: "bcd3c8d8-2c37-43ec-8cab-ed05b059bb75") : secret "insights-runtime-extractor-tls" not found Sep 09 17:21:04.378031 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:04.377959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:21:08.911212 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:08.911128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:21:08.911212 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:08.911195 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:21:08.911792 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:08.911286 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 17:21:08.911792 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:08.911355 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls podName:930aa645-7318-4c90-850d-560c1430ab0f nodeName:}" failed. No retries permitted until 2026-09-09 17:21:24.911339436 +0000 UTC m=+279.259700888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nh2hz" (UID: "930aa645-7318-4c90-850d-560c1430ab0f") : secret "cluster-monitoring-operator-tls" not found Sep 09 17:21:08.913476 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:08.913457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/810557d8-1bd0-4e5f-a5f8-af18803cb208-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-fks5x\" (UID: \"810557d8-1bd0-4e5f-a5f8-af18803cb208\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:21:09.006467 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:09.006434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" Sep 09 17:21:09.271164 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:09.271141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x"] Sep 09 17:21:09.872271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:09.872226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" event={"ID":"810557d8-1bd0-4e5f-a5f8-af18803cb208","Type":"ContainerStarted","Data":"7188400b17da180832edd058a27c8f325cb3c2534440e6b5b403e280fcaa72e8"} Sep 09 17:21:11.880516 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:11.880483 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" event={"ID":"810557d8-1bd0-4e5f-a5f8-af18803cb208","Type":"ContainerStarted","Data":"bf14e62f6f399e3129bf1592f19c68d600a2510b09367938401ea77e364d58d8"} Sep 09 17:21:11.880983 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:11.880563 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" event={"ID":"810557d8-1bd0-4e5f-a5f8-af18803cb208","Type":"ContainerStarted","Data":"ce18c1ea237591d5ae11865fde7f4aab27e458e62a502930adcb0886e642722e"} Sep 09 17:21:11.895672 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:11.895604 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-fks5x" podStartSLOduration=17.314053136 podStartE2EDuration="18.895589117s" totalImagesPullingTime="1.581535981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:11.894431115 +0000 UTC m=+266.242792587" watchObservedRunningTime="2026-09-09 17:21:11.895589117 +0000 UTC m=+266.243950590" Sep 09 17:21:12.437590 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:12.437549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:21:12.439704 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:12.439681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bcd3c8d8-2c37-43ec-8cab-ed05b059bb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wjqgn\" (UID: \"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75\") " pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:21:12.612898 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:12.612867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wjqgn" Sep 09 17:21:12.880386 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:12.880274 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wjqgn"] Sep 09 17:21:12.882960 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:12.882918 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd3c8d8_2c37_43ec_8cab_ed05b059bb75.slice/crio-afafd6f7b40ef4d0e83f2ede0606eb4de7ca1a5e88da9b63f05fd8e62a05edb7 WatchSource:0}: Error finding container afafd6f7b40ef4d0e83f2ede0606eb4de7ca1a5e88da9b63f05fd8e62a05edb7: Status 404 returned error can't find the container with id afafd6f7b40ef4d0e83f2ede0606eb4de7ca1a5e88da9b63f05fd8e62a05edb7 Sep 09 17:21:13.888343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:13.888260 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wjqgn" event={"ID":"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75","Type":"ContainerStarted","Data":"fd68f7778e5123af70afcdee77ae2d474f4c8d2f29c8fcb76809e8d0d3c966e8"} Sep 09 17:21:13.888343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:13.888317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wjqgn" event={"ID":"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75","Type":"ContainerStarted","Data":"b89228a22c817cccde90bbb52361707ee7f3809ed28b39f3bac7a659828b22d3"} Sep 09 17:21:13.888343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:13.888335 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wjqgn" event={"ID":"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75","Type":"ContainerStarted","Data":"afafd6f7b40ef4d0e83f2ede0606eb4de7ca1a5e88da9b63f05fd8e62a05edb7"} Sep 09 17:21:15.896153 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:15.896117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wjqgn" event={"ID":"bcd3c8d8-2c37-43ec-8cab-ed05b059bb75","Type":"ContainerStarted","Data":"28ff943b8d21c0a7e0b2137eb2a6b8e42f5c4cb4e3a4d4d6687884788f627bd4"} Sep 09 17:21:15.914462 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:15.914414 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wjqgn" podStartSLOduration=18.040879111 podStartE2EDuration="19.914399336s" totalImagesPullingTime="1.873520225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:20:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:15.913614027 +0000 UTC m=+270.261975501" watchObservedRunningTime="2026-09-09 17:21:15.914399336 +0000 UTC m=+270.262760808" Sep 09 17:21:19.125311 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:19.125279 2580 scope.go:117] "RemoveContainer" containerID="20bc98b7e3038a3367fde8fac97c3aad568484e7d699f15b3bdddf000fc9111a" Sep 09 17:21:19.912930 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:19.912898 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:21:19.913099 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:19.912960 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-z9596" event={"ID":"9b4e0a01-e009-4a59-9826-b82707527015","Type":"ContainerStarted","Data":"0d7304353214f82516ba355010bec103d0124e9afbe7a320bee096511f1ce939"} Sep 09 17:21:19.913327 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:19.913300 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:21:19.920652 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:19.920620 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-z9596" Sep 09 17:21:19.932052 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:19.932002 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-z9596" podStartSLOduration=24.945064869 podStartE2EDuration="27.931984522s" totalImagesPullingTime="2.986919653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:19.931337011 +0000 UTC m=+274.279698483" watchObservedRunningTime="2026-09-09 17:21:19.931984522 +0000 UTC m=+274.280345995" Sep 09 17:21:20.062642 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.062606 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-2nnll"] Sep 09 17:21:20.065757 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.065740 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-2nnll" Sep 09 17:21:20.068249 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.068231 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 17:21:20.068348 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.068252 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-rxqh4\"" Sep 09 17:21:20.068348 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.068252 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 17:21:20.077357 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.077335 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-2nnll"] Sep 09 17:21:20.200019 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.199935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmmxk\" (UniqueName: \"kubernetes.io/projected/0f4ede09-9816-432e-9f1f-652dd1ab3a5f-kube-api-access-tmmxk\") pod \"downloads-85d98c998f-2nnll\" (UID: \"0f4ede09-9816-432e-9f1f-652dd1ab3a5f\") " pod="openshift-console/downloads-85d98c998f-2nnll" Sep 09 17:21:20.300432 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.300395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmmxk\" (UniqueName: \"kubernetes.io/projected/0f4ede09-9816-432e-9f1f-652dd1ab3a5f-kube-api-access-tmmxk\") pod \"downloads-85d98c998f-2nnll\" (UID: \"0f4ede09-9816-432e-9f1f-652dd1ab3a5f\") " pod="openshift-console/downloads-85d98c998f-2nnll" Sep 09 17:21:20.309726 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.309682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmmxk\" (UniqueName: \"kubernetes.io/projected/0f4ede09-9816-432e-9f1f-652dd1ab3a5f-kube-api-access-tmmxk\") pod \"downloads-85d98c998f-2nnll\" (UID: \"0f4ede09-9816-432e-9f1f-652dd1ab3a5f\") " pod="openshift-console/downloads-85d98c998f-2nnll" Sep 09 17:21:20.373822 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.373790 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-2nnll" Sep 09 17:21:20.642703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.640919 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-2nnll"] Sep 09 17:21:20.646435 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:20.646407 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4ede09_9816_432e_9f1f_652dd1ab3a5f.slice/crio-fb5a6c0014bf557600d9b56a221f0ba48eb75840f74ce49609bce3a6f6d691b5 WatchSource:0}: Error finding container fb5a6c0014bf557600d9b56a221f0ba48eb75840f74ce49609bce3a6f6d691b5: Status 404 returned error can't find the container with id fb5a6c0014bf557600d9b56a221f0ba48eb75840f74ce49609bce3a6f6d691b5 Sep 09 17:21:20.917438 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:20.917354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-2nnll" event={"ID":"0f4ede09-9816-432e-9f1f-652dd1ab3a5f","Type":"ContainerStarted","Data":"fb5a6c0014bf557600d9b56a221f0ba48eb75840f74ce49609bce3a6f6d691b5"} Sep 09 17:21:24.941189 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:24.941152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:21:24.943811 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:24.943780 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/930aa645-7318-4c90-850d-560c1430ab0f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nh2hz\" (UID: \"930aa645-7318-4c90-850d-560c1430ab0f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:21:25.106241 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:25.106201 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" Sep 09 17:21:25.384344 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:25.384309 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz"] Sep 09 17:21:25.387673 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:25.387648 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930aa645_7318_4c90_850d_560c1430ab0f.slice/crio-d12098d1bb996b4a58614337ec563f097f49118c24dbe641c826be21e5bbd29c WatchSource:0}: Error finding container d12098d1bb996b4a58614337ec563f097f49118c24dbe641c826be21e5bbd29c: Status 404 returned error can't find the container with id d12098d1bb996b4a58614337ec563f097f49118c24dbe641c826be21e5bbd29c Sep 09 17:21:25.936664 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:25.936625 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" event={"ID":"930aa645-7318-4c90-850d-560c1430ab0f","Type":"ContainerStarted","Data":"d12098d1bb996b4a58614337ec563f097f49118c24dbe641c826be21e5bbd29c"} Sep 09 17:21:26.592329 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:26.592283 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" podUID="daee458d-204d-4371-898f-5094dfe79c09" Sep 09 17:21:26.592329 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:26.592283 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8j8h9" podUID="b79ceb92-b128-4b3b-9fa5-64ae8bbb384f" Sep 09 17:21:26.946199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:26.942645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8j8h9" Sep 09 17:21:26.946199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:26.943118 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:21:27.936827 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:27.936794 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6"] Sep 09 17:21:27.940180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:27.940157 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" Sep 09 17:21:27.943223 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:27.943053 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 17:21:27.943223 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:27.943058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vg8dx\"" Sep 09 17:21:27.948821 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:27.948788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" event={"ID":"930aa645-7318-4c90-850d-560c1430ab0f","Type":"ContainerStarted","Data":"4d33a23d24d19804d7f3ae3cccae095e1d5cf2b0f248e624093c0b6fd7cf9741"} Sep 09 17:21:27.954093 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:27.954065 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6"] Sep 09 17:21:27.998634 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:27.998569 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nh2hz" podStartSLOduration=33.969597582 podStartE2EDuration="35.998546098s" totalImagesPullingTime="2.028948516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:20:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:27.997101701 +0000 UTC m=+282.345463174" watchObservedRunningTime="2026-09-09 17:21:27.998546098 +0000 UTC m=+282.346907571" Sep 09 17:21:28.068794 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:28.068757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/45c9e5f4-26c4-40f4-8bae-e52c0271909a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zrqk6\" (UID: \"45c9e5f4-26c4-40f4-8bae-e52c0271909a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" Sep 09 17:21:28.170147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:28.170105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/45c9e5f4-26c4-40f4-8bae-e52c0271909a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zrqk6\" (UID: \"45c9e5f4-26c4-40f4-8bae-e52c0271909a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" Sep 09 17:21:28.170326 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:28.170255 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 09 17:21:28.170372 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:28.170328 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45c9e5f4-26c4-40f4-8bae-e52c0271909a-tls-certificates podName:45c9e5f4-26c4-40f4-8bae-e52c0271909a nodeName:}" failed. No retries permitted until 2026-09-09 17:21:28.670309766 +0000 UTC m=+283.018671222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/45c9e5f4-26c4-40f4-8bae-e52c0271909a-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-zrqk6" (UID: "45c9e5f4-26c4-40f4-8bae-e52c0271909a") : secret "prometheus-operator-admission-webhook-tls" not found Sep 09 17:21:28.674868 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:28.674823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/45c9e5f4-26c4-40f4-8bae-e52c0271909a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zrqk6\" (UID: \"45c9e5f4-26c4-40f4-8bae-e52c0271909a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" Sep 09 17:21:28.677514 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:28.677486 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/45c9e5f4-26c4-40f4-8bae-e52c0271909a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zrqk6\" (UID: \"45c9e5f4-26c4-40f4-8bae-e52c0271909a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" Sep 09 17:21:28.851268 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:28.851220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" Sep 09 17:21:29.000058 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:28.999970 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84dc44b9d6-wg8np"] Sep 09 17:21:29.004255 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.004226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.006862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.006836 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 17:21:29.008005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.007982 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 17:21:29.008005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.007998 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 17:21:29.008186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.008029 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 17:21:29.008186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.008118 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 17:21:29.008186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.008128 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-rfpcp\"" Sep 09 17:21:29.011321 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.011296 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84dc44b9d6-wg8np"] Sep 09 17:21:29.079308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.079263 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-service-ca\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.079494 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.079338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-oauth-serving-cert\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.079494 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.079372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-serving-cert\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.079494 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.079404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-oauth-config\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.079604 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.079508 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swx27\" (UniqueName: \"kubernetes.io/projected/defe28f2-e89b-4223-ac1b-03f33831140e-kube-api-access-swx27\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.079604 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.079572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-console-config\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.124310 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.124280 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6"] Sep 09 17:21:29.127151 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:29.127126 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c9e5f4_26c4_40f4_8bae_e52c0271909a.slice/crio-6087b85d715660fdfb19d1480409c2a736e2ca80c70634ceb9ce569962402bcb WatchSource:0}: Error finding container 6087b85d715660fdfb19d1480409c2a736e2ca80c70634ceb9ce569962402bcb: Status 404 returned error can't find the container with id 6087b85d715660fdfb19d1480409c2a736e2ca80c70634ceb9ce569962402bcb Sep 09 17:21:29.180518 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.180476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-service-ca\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.180701 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.180542 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-oauth-serving-cert\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.180701 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.180576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-serving-cert\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.180701 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.180609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-oauth-config\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.180701 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.180653 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swx27\" (UniqueName: \"kubernetes.io/projected/defe28f2-e89b-4223-ac1b-03f33831140e-kube-api-access-swx27\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.180956 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.180734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-console-config\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.181352 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.181324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-service-ca\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.181557 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.181534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-console-config\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.181655 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.181632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-oauth-serving-cert\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.183305 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.183284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-oauth-config\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.183530 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.183510 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-serving-cert\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.191006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.190987 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swx27\" (UniqueName: \"kubernetes.io/projected/defe28f2-e89b-4223-ac1b-03f33831140e-kube-api-access-swx27\") pod \"console-84dc44b9d6-wg8np\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.316556 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.316514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:29.591591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.591520 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84dc44b9d6-wg8np"] Sep 09 17:21:29.594837 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:29.594809 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefe28f2_e89b_4223_ac1b_03f33831140e.slice/crio-53a2cc8c465c84428caa64c4662163964a2dcf031cd59cd7a109679af60ab63e WatchSource:0}: Error finding container 53a2cc8c465c84428caa64c4662163964a2dcf031cd59cd7a109679af60ab63e: Status 404 returned error can't find the container with id 53a2cc8c465c84428caa64c4662163964a2dcf031cd59cd7a109679af60ab63e Sep 09 17:21:29.958882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.958788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dc44b9d6-wg8np" event={"ID":"defe28f2-e89b-4223-ac1b-03f33831140e","Type":"ContainerStarted","Data":"53a2cc8c465c84428caa64c4662163964a2dcf031cd59cd7a109679af60ab63e"} Sep 09 17:21:29.960363 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.960334 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" event={"ID":"45c9e5f4-26c4-40f4-8bae-e52c0271909a","Type":"ContainerStarted","Data":"6087b85d715660fdfb19d1480409c2a736e2ca80c70634ceb9ce569962402bcb"} Sep 09 17:21:29.987789 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.987757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:21:29.987925 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.987811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:21:29.990297 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.990272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b79ceb92-b128-4b3b-9fa5-64ae8bbb384f-metrics-tls\") pod \"dns-default-8j8h9\" (UID: \"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f\") " pod="openshift-dns/dns-default-8j8h9" Sep 09 17:21:29.990499 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:29.990481 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"image-registry-565b7696cc-fptf6\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:21:30.248007 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:30.247933 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cblwm\"" Sep 09 17:21:30.248007 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:30.247933 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kdgth\"" Sep 09 17:21:30.254106 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:30.254078 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:21:30.254253 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:30.254128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8j8h9" Sep 09 17:21:37.536446 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.536363 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565b7696cc-fptf6"] Sep 09 17:21:37.538025 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.537996 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8j8h9"] Sep 09 17:21:37.541022 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:37.540946 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaee458d_204d_4371_898f_5094dfe79c09.slice/crio-bd4917f71e015469ff669471594bb15104d377947b4c96209e9dbc691ec43771 WatchSource:0}: Error finding container bd4917f71e015469ff669471594bb15104d377947b4c96209e9dbc691ec43771: Status 404 returned error can't find the container with id bd4917f71e015469ff669471594bb15104d377947b4c96209e9dbc691ec43771 Sep 09 17:21:37.541666 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:37.541624 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79ceb92_b128_4b3b_9fa5_64ae8bbb384f.slice/crio-f3e675fc597704848b2bc3dc556897c2959506e0e7b2599c8606f9c541af6aea WatchSource:0}: Error finding container f3e675fc597704848b2bc3dc556897c2959506e0e7b2599c8606f9c541af6aea: Status 404 returned error can't find the container with id f3e675fc597704848b2bc3dc556897c2959506e0e7b2599c8606f9c541af6aea Sep 09 17:21:37.993747 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.992584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-2nnll" event={"ID":"0f4ede09-9816-432e-9f1f-652dd1ab3a5f","Type":"ContainerStarted","Data":"322358196b87abf66cb9111194a3582c48b6c8c4a61985772e463d7dc4d74121"} Sep 09 17:21:37.993747 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.993597 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-2nnll" Sep 09 17:21:37.996091 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.996065 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" event={"ID":"daee458d-204d-4371-898f-5094dfe79c09","Type":"ContainerStarted","Data":"ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c"} Sep 09 17:21:37.996229 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.996102 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" event={"ID":"daee458d-204d-4371-898f-5094dfe79c09","Type":"ContainerStarted","Data":"bd4917f71e015469ff669471594bb15104d377947b4c96209e9dbc691ec43771"} Sep 09 17:21:37.996297 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.996272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:21:37.998048 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.998020 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8j8h9" event={"ID":"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f","Type":"ContainerStarted","Data":"f3e675fc597704848b2bc3dc556897c2959506e0e7b2599c8606f9c541af6aea"} Sep 09 17:21:37.999927 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:37.999860 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" event={"ID":"45c9e5f4-26c4-40f4-8bae-e52c0271909a","Type":"ContainerStarted","Data":"0736046fe041f1474acc727befe27e051557f3e8c88dbdb3101d96200727279d"} Sep 09 17:21:38.000334 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:38.000290 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" Sep 09 17:21:38.006130 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:38.006103 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" Sep 09 17:21:38.009228 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:38.009200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-2nnll" Sep 09 17:21:38.009762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:38.009697 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-2nnll" podStartSLOduration=1.337463628 podStartE2EDuration="18.00967084s" totalImagesPullingTime="16.672207212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:38.008531137 +0000 UTC m=+292.356892637" watchObservedRunningTime="2026-09-09 17:21:38.00967084 +0000 UTC m=+292.358032313" Sep 09 17:21:38.030921 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:38.030824 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zrqk6" podStartSLOduration=2.896165486 podStartE2EDuration="11.03080456s" totalImagesPullingTime="8.134639074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:38.029786776 +0000 UTC m=+292.378148251" watchObservedRunningTime="2026-09-09 17:21:38.03080456 +0000 UTC m=+292.379166033" Sep 09 17:21:38.055552 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:38.055479 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" podStartSLOduration=291.055456482 podStartE2EDuration="4m51.055456482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:38.054073121 +0000 UTC m=+292.402434618" watchObservedRunningTime="2026-09-09 17:21:38.055456482 +0000 UTC m=+292.403817956" Sep 09 17:21:39.128000 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:21:39.127948 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kkvpl" podUID="f9649a0b-cc7d-4eb0-9952-865e10002bd7" Sep 09 17:21:42.021474 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:42.021434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8j8h9" event={"ID":"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f","Type":"ContainerStarted","Data":"d720a767de393a9e35f1e84d550f8745f36b2d4741bc5211914a3a6516613dae"} Sep 09 17:21:42.022043 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:42.021507 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8j8h9" event={"ID":"b79ceb92-b128-4b3b-9fa5-64ae8bbb384f","Type":"ContainerStarted","Data":"473e55cac6b8e3661cb7afb9237871eddde405e0e9bb77907d1bab8be4ce00e3"} Sep 09 17:21:42.022043 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:42.021537 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8j8h9" Sep 09 17:21:42.023263 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:42.023237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dc44b9d6-wg8np" event={"ID":"defe28f2-e89b-4223-ac1b-03f33831140e","Type":"ContainerStarted","Data":"28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7"} Sep 09 17:21:42.037973 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:42.037931 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8j8h9" podStartSLOduration=259.131676503 podStartE2EDuration="4m23.03791433s" totalImagesPullingTime="3.906237827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:42.036694229 +0000 UTC m=+296.385055708" watchObservedRunningTime="2026-09-09 17:21:42.03791433 +0000 UTC m=+296.386275803" Sep 09 17:21:42.053050 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:42.053004 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84dc44b9d6-wg8np" podStartSLOduration=2.192329373 podStartE2EDuration="14.052988812s" totalImagesPullingTime="11.860659439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:42.051823791 +0000 UTC m=+296.400185267" watchObservedRunningTime="2026-09-09 17:21:42.052988812 +0000 UTC m=+296.401350285" Sep 09 17:21:44.319591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.319557 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd"] Sep 09 17:21:44.355043 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.355012 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd"] Sep 09 17:21:44.355215 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.355063 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn"] Sep 09 17:21:44.355215 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.355110 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.357591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.357566 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 17:21:44.357760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.357620 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 17:21:44.358006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.357989 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 17:21:44.358210 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.358193 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dt6lx\"" Sep 09 17:21:44.378968 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.378939 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4dqph"] Sep 09 17:21:44.379097 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.379066 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.382380 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.382353 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 09 17:21:44.382496 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.382403 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 09 17:21:44.382496 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.382406 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p48bv\"" Sep 09 17:21:44.382615 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.382494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 09 17:21:44.404153 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.404124 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn"] Sep 09 17:21:44.404289 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.404247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.407020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.406995 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 17:21:44.407150 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.407047 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 17:21:44.407150 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.407056 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4v4kg\"" Sep 09 17:21:44.407310 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.407214 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 17:21:44.524870 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.524835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-sys\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525053 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96f1a4da-722a-4a7f-913d-e03bfd93f51a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.525053 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-accelerators-collector-config\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525178 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96f1a4da-722a-4a7f-913d-e03bfd93f51a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.525178 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de31f071-6ade-4899-a46a-7ed675ee649b-metrics-client-ca\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525178 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525155 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.525318 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-tls\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525318 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-root\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525318 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.525318 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.525506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525338 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l978b\" (UniqueName: \"kubernetes.io/projected/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-api-access-l978b\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.525506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-textfile\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-wtmp\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcg5r\" (UniqueName: \"kubernetes.io/projected/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-kube-api-access-pcg5r\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.525506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525477 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlcsv\" (UniqueName: \"kubernetes.io/projected/de31f071-6ade-4899-a46a-7ed675ee649b-kube-api-access-zlcsv\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.525506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.525862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.525862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.525562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.627096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.626998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-sys\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.627096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96f1a4da-722a-4a7f-913d-e03bfd93f51a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.627096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-accelerators-collector-config\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.627387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627124 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-sys\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.627387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96f1a4da-722a-4a7f-913d-e03bfd93f51a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.627387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de31f071-6ade-4899-a46a-7ed675ee649b-metrics-client-ca\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.627387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.627387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627268 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-tls\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.627387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-root\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.627387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.627387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.627797 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-root\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.627964 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de31f071-6ade-4899-a46a-7ed675ee649b-metrics-client-ca\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.627964 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627950 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96f1a4da-722a-4a7f-913d-e03bfd93f51a-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.628149 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-accelerators-collector-config\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.628149 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.627425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l978b\" (UniqueName: \"kubernetes.io/projected/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-api-access-l978b\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.628149 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-textfile\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.628149 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-wtmp\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.628149 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcg5r\" (UniqueName: \"kubernetes.io/projected/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-kube-api-access-pcg5r\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.628390 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/96f1a4da-722a-4a7f-913d-e03bfd93f51a-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.628390 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-wtmp\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.628390 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628380 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-textfile\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.628587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628439 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.628587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlcsv\" (UniqueName: \"kubernetes.io/projected/de31f071-6ade-4899-a46a-7ed675ee649b-kube-api-access-zlcsv\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.628587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628513 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.628587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.628587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.628866 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.628677 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.629526 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.629484 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.630845 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.630800 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.631699 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.631655 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.631839 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.631816 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.631911 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.631894 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.632065 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.632038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.632639 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.632618 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/de31f071-6ade-4899-a46a-7ed675ee649b-node-exporter-tls\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.635397 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.635374 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l978b\" (UniqueName: \"kubernetes.io/projected/96f1a4da-722a-4a7f-913d-e03bfd93f51a-kube-api-access-l978b\") pod \"kube-state-metrics-74cb57df7-h5zqn\" (UID: \"96f1a4da-722a-4a7f-913d-e03bfd93f51a\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.636812 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.636779 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcg5r\" (UniqueName: \"kubernetes.io/projected/bebc1406-5ed1-4286-9f7f-3cb9158e09b7-kube-api-access-pcg5r\") pod \"openshift-state-metrics-657fd6c54-2jrfd\" (UID: \"bebc1406-5ed1-4286-9f7f-3cb9158e09b7\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.637325 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.637300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlcsv\" (UniqueName: \"kubernetes.io/projected/de31f071-6ade-4899-a46a-7ed675ee649b-kube-api-access-zlcsv\") pod \"node-exporter-4dqph\" (UID: \"de31f071-6ade-4899-a46a-7ed675ee649b\") " pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.666202 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.666171 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" Sep 09 17:21:44.689374 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.689340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" Sep 09 17:21:44.714370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.714337 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4dqph" Sep 09 17:21:44.979102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:44.979075 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd"] Sep 09 17:21:44.981263 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:44.981228 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebc1406_5ed1_4286_9f7f_3cb9158e09b7.slice/crio-99c4558ec2d201cb0d002d8ff01cef2a54f88935ea5194e7c3f35c42dad6540c WatchSource:0}: Error finding container 99c4558ec2d201cb0d002d8ff01cef2a54f88935ea5194e7c3f35c42dad6540c: Status 404 returned error can't find the container with id 99c4558ec2d201cb0d002d8ff01cef2a54f88935ea5194e7c3f35c42dad6540c Sep 09 17:21:45.012965 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:45.012931 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn"] Sep 09 17:21:45.015625 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:45.015595 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f1a4da_722a_4a7f_913d_e03bfd93f51a.slice/crio-0b31aa8508056851c3f7e088ac898d5b85ee2e5f4b30449cb9cc88bbc678cf60 WatchSource:0}: Error finding container 0b31aa8508056851c3f7e088ac898d5b85ee2e5f4b30449cb9cc88bbc678cf60: Status 404 returned error can't find the container with id 0b31aa8508056851c3f7e088ac898d5b85ee2e5f4b30449cb9cc88bbc678cf60 Sep 09 17:21:45.039351 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:45.039323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" event={"ID":"bebc1406-5ed1-4286-9f7f-3cb9158e09b7","Type":"ContainerStarted","Data":"99c4558ec2d201cb0d002d8ff01cef2a54f88935ea5194e7c3f35c42dad6540c"} Sep 09 17:21:45.040736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:45.040657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" event={"ID":"96f1a4da-722a-4a7f-913d-e03bfd93f51a","Type":"ContainerStarted","Data":"0b31aa8508056851c3f7e088ac898d5b85ee2e5f4b30449cb9cc88bbc678cf60"} Sep 09 17:21:45.041962 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:45.041937 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dqph" event={"ID":"de31f071-6ade-4899-a46a-7ed675ee649b","Type":"ContainerStarted","Data":"c47fc3c0ee024db7a2ebe25882c65c31c28a5ba0825f81094a455165497b761f"} Sep 09 17:21:46.054007 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:46.053900 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" event={"ID":"bebc1406-5ed1-4286-9f7f-3cb9158e09b7","Type":"ContainerStarted","Data":"01bd7887bc5f28c43446ba8592e4a55ef74b2bb7db644aa0225e3bb7092654ea"} Sep 09 17:21:46.054007 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:46.053970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" event={"ID":"bebc1406-5ed1-4286-9f7f-3cb9158e09b7","Type":"ContainerStarted","Data":"f3e2ddc53efba78764da33d852ded42c28ffa87a36df796cd89ae87b22002590"} Sep 09 17:21:46.111284 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:46.111250 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:21:46.111486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:46.111468 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:21:46.127871 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:46.127749 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:21:46.131777 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:46.131755 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:21:46.151050 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:46.150938 2580 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 17:21:47.308304 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.308274 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c85b85d77-lpq9q"] Sep 09 17:21:47.331175 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.331148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c85b85d77-lpq9q"] Sep 09 17:21:47.331318 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.331296 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.334083 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.334062 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 09 17:21:47.334200 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.334068 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-8l5nb\"" Sep 09 17:21:47.334200 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.334170 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 09 17:21:47.334300 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.334206 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d5u1lill5ogp7\"" Sep 09 17:21:47.334436 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.334415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 09 17:21:47.334496 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.334477 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 09 17:21:47.334567 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.334422 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 09 17:21:47.354182 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.354157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkzd\" (UniqueName: \"kubernetes.io/projected/8b58e555-f3b9-42cc-9e26-e822205efaf0-kube-api-access-lfkzd\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.354291 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.354187 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b58e555-f3b9-42cc-9e26-e822205efaf0-metrics-client-ca\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.354291 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.354210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-grpc-tls\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.354291 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.354254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.354442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.354307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.354442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.354396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.354518 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.354436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.354518 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.354487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-tls\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.455468 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.455432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkzd\" (UniqueName: \"kubernetes.io/projected/8b58e555-f3b9-42cc-9e26-e822205efaf0-kube-api-access-lfkzd\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.455633 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.455476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b58e555-f3b9-42cc-9e26-e822205efaf0-metrics-client-ca\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.455633 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.455519 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-grpc-tls\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.455633 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.455552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.455633 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.455618 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.455877 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.455666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.455877 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.455823 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.455877 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.455868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-tls\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.456447 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.456377 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b58e555-f3b9-42cc-9e26-e822205efaf0-metrics-client-ca\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.458224 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.458194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-grpc-tls\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.458326 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.458262 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-tls\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.458441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.458417 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.458515 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.458492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.458586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.458563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.458638 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.458586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b58e555-f3b9-42cc-9e26-e822205efaf0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.463029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.463009 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkzd\" (UniqueName: \"kubernetes.io/projected/8b58e555-f3b9-42cc-9e26-e822205efaf0-kube-api-access-lfkzd\") pod \"thanos-querier-c85b85d77-lpq9q\" (UID: \"8b58e555-f3b9-42cc-9e26-e822205efaf0\") " pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:47.640593 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:47.640563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:48.016491 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.016445 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c85b85d77-lpq9q"] Sep 09 17:21:48.037839 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:48.037814 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b58e555_f3b9_42cc_9e26_e822205efaf0.slice/crio-cba8bb11441088469c9e7756fd2c7adc8cb998015e58d886fcf036994e751888 WatchSource:0}: Error finding container cba8bb11441088469c9e7756fd2c7adc8cb998015e58d886fcf036994e751888: Status 404 returned error can't find the container with id cba8bb11441088469c9e7756fd2c7adc8cb998015e58d886fcf036994e751888 Sep 09 17:21:48.040233 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.040216 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:21:48.065215 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.065151 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" event={"ID":"bebc1406-5ed1-4286-9f7f-3cb9158e09b7","Type":"ContainerStarted","Data":"55d2f36e89d34e039a1c3a4aedb41e96029704cf5e43e0241c2247fe2b5d7431"} Sep 09 17:21:48.067148 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.067126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" event={"ID":"96f1a4da-722a-4a7f-913d-e03bfd93f51a","Type":"ContainerStarted","Data":"8a6b865a53e0ca0d2838c04dcc0a353e770ef531b41c70eaa06c229f61a377ef"} Sep 09 17:21:48.067243 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.067178 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" event={"ID":"96f1a4da-722a-4a7f-913d-e03bfd93f51a","Type":"ContainerStarted","Data":"9f31d90df155a760074be8412c8546ac218164a3fb2f8f25f7cdfb950ca391e8"} Sep 09 17:21:48.068788 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.068763 2580 generic.go:356] "Generic (PLEG): container finished" podID="de31f071-6ade-4899-a46a-7ed675ee649b" containerID="25234e1865c7ca1e564e7bcac3bf0d96abcdad9ec614d3d2abefb6f771f5c2d8" exitCode=0 Sep 09 17:21:48.068884 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.068845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dqph" event={"ID":"de31f071-6ade-4899-a46a-7ed675ee649b","Type":"ContainerDied","Data":"25234e1865c7ca1e564e7bcac3bf0d96abcdad9ec614d3d2abefb6f771f5c2d8"} Sep 09 17:21:48.070319 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.070291 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" event={"ID":"8b58e555-f3b9-42cc-9e26-e822205efaf0","Type":"ContainerStarted","Data":"cba8bb11441088469c9e7756fd2c7adc8cb998015e58d886fcf036994e751888"} Sep 09 17:21:48.082104 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:48.082063 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-2jrfd" podStartSLOduration=1.609667709 podStartE2EDuration="4.082049732s" totalImagesPullingTime="2.472382023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:48.080973417 +0000 UTC m=+302.429334890" watchObservedRunningTime="2026-09-09 17:21:48.082049732 +0000 UTC m=+302.430411205" Sep 09 17:21:49.078888 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.078851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" event={"ID":"96f1a4da-722a-4a7f-913d-e03bfd93f51a","Type":"ContainerStarted","Data":"e7dcb72b8686d257016e9884fa0d75cb6ad76c968bf59814a3cb18ce1767a4ff"} Sep 09 17:21:49.083148 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.082609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dqph" event={"ID":"de31f071-6ade-4899-a46a-7ed675ee649b","Type":"ContainerStarted","Data":"a6840f5df5b34e9a096f963b0033ed452a6886d8bef6a1480dd9eaca28e8926d"} Sep 09 17:21:49.083148 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.082673 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4dqph" event={"ID":"de31f071-6ade-4899-a46a-7ed675ee649b","Type":"ContainerStarted","Data":"d5bcbe0414241767d998fca50ee4c3f00b5ebd4441b7754f6d048ff5b27ea8d6"} Sep 09 17:21:49.098560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.098507 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h5zqn" podStartSLOduration=2.37028511 podStartE2EDuration="5.098488019s" totalImagesPullingTime="2.728202909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:49.096481074 +0000 UTC m=+303.444842547" watchObservedRunningTime="2026-09-09 17:21:49.098488019 +0000 UTC m=+303.446849491" Sep 09 17:21:49.118603 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.118559 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4dqph" podStartSLOduration=2.05134103 podStartE2EDuration="5.118543562s" totalImagesPullingTime="2.967137297s" totalInitContainerRuntime="100.065235ms" isStatefulPod=true podCreationTimestamp="2026-09-09 17:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:49.117813632 +0000 UTC m=+303.466175106" watchObservedRunningTime="2026-09-09 17:21:49.118543562 +0000 UTC m=+303.466905037" Sep 09 17:21:49.317245 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.317209 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:49.317471 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.317275 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:21:49.318613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.318588 2580 patch_prober.go:28] interesting pod/console-84dc44b9d6-wg8np container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused" start-of-body= Sep 09 17:21:49.318680 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:49.318632 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-84dc44b9d6-wg8np" podUID="defe28f2-e89b-4223-ac1b-03f33831140e" containerName="console" probeResult="failure" output="Get \"https://10.132.0.19:8443/health\": dial tcp 10.132.0.19:8443: connect: connection refused" Sep 09 17:21:50.124752 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.124704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:21:50.259812 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.259787 2580 patch_prober.go:28] interesting pod/image-registry-565b7696cc-fptf6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 17:21:50.259919 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.259829 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" podUID="daee458d-204d-4371-898f-5094dfe79c09" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 17:21:50.312599 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.312560 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84dc44b9d6-wg8np"] Sep 09 17:21:50.337946 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.337920 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fb79c4584-r96kj"] Sep 09 17:21:50.358952 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.358928 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fb79c4584-r96kj"] Sep 09 17:21:50.359081 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.359058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.365869 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.365736 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 17:21:50.383962 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.383884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-oauth-config\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.383962 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.383923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-trusted-ca-bundle\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.384161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.383977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-serving-cert\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.384161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.384052 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-console-config\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.384161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.384077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-service-ca\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.384161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.384136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdh5\" (UniqueName: \"kubernetes.io/projected/0abada0d-d939-41fc-b683-6054ef953868-kube-api-access-btdh5\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.384377 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.384205 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-oauth-serving-cert\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.485148 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-oauth-serving-cert\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.485271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-oauth-config\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.485271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-trusted-ca-bundle\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.485271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-serving-cert\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.485423 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-console-config\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.485423 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-service-ca\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.485514 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdh5\" (UniqueName: \"kubernetes.io/projected/0abada0d-d939-41fc-b683-6054ef953868-kube-api-access-btdh5\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.485933 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-oauth-serving-cert\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.486024 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.485999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-service-ca\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.486064 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.486047 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-console-config\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.486097 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.486072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-trusted-ca-bundle\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.487532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.487517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-serving-cert\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.487731 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.487691 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-oauth-config\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.492636 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.492604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdh5\" (UniqueName: \"kubernetes.io/projected/0abada0d-d939-41fc-b683-6054ef953868-kube-api-access-btdh5\") pod \"console-7fb79c4584-r96kj\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.668431 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.668353 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:21:50.961876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:50.961843 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fb79c4584-r96kj"] Sep 09 17:21:50.964166 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:21:50.964125 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0abada0d_d939_41fc_b683_6054ef953868.slice/crio-a7095fb9d32bb8a64f562b3b410d90b40bd90f5049e3f66f1f44e19014cef204 WatchSource:0}: Error finding container a7095fb9d32bb8a64f562b3b410d90b40bd90f5049e3f66f1f44e19014cef204: Status 404 returned error can't find the container with id a7095fb9d32bb8a64f562b3b410d90b40bd90f5049e3f66f1f44e19014cef204 Sep 09 17:21:51.093032 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:51.093000 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" event={"ID":"8b58e555-f3b9-42cc-9e26-e822205efaf0","Type":"ContainerStarted","Data":"3fac8327056e2f143787cd4ad8374c944079c8e1d1017deae001ae63f900c32e"} Sep 09 17:21:51.093143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:51.093084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" event={"ID":"8b58e555-f3b9-42cc-9e26-e822205efaf0","Type":"ContainerStarted","Data":"e69148ff7ca021725bcad975a6919ffe8580b8eeba3ba6f740599e20a9d04280"} Sep 09 17:21:51.093143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:51.093127 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" event={"ID":"8b58e555-f3b9-42cc-9e26-e822205efaf0","Type":"ContainerStarted","Data":"c5698ff0d0c8ff205b48863727f6caceca0d877ef8c451822e9b7a069e652fda"} Sep 09 17:21:51.094595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:51.094569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb79c4584-r96kj" event={"ID":"0abada0d-d939-41fc-b683-6054ef953868","Type":"ContainerStarted","Data":"aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd"} Sep 09 17:21:51.094688 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:51.094604 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb79c4584-r96kj" event={"ID":"0abada0d-d939-41fc-b683-6054ef953868","Type":"ContainerStarted","Data":"a7095fb9d32bb8a64f562b3b410d90b40bd90f5049e3f66f1f44e19014cef204"} Sep 09 17:21:52.031424 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:52.031400 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8j8h9" Sep 09 17:21:52.102070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:52.102028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" event={"ID":"8b58e555-f3b9-42cc-9e26-e822205efaf0","Type":"ContainerStarted","Data":"3a112f448da3926fb2addb9aa3664882ceb7872a896eede474aa202b1a0cd055"} Sep 09 17:21:52.102209 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:52.102096 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" event={"ID":"8b58e555-f3b9-42cc-9e26-e822205efaf0","Type":"ContainerStarted","Data":"bcd85a0b3997439251109aa3e68da8ec4c246384477b90d67b1908d5715ef2fa"} Sep 09 17:21:52.135781 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:52.135738 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fb79c4584-r96kj" podStartSLOduration=2.135705643 podStartE2EDuration="2.135705643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:52.134570354 +0000 UTC m=+306.482931840" watchObservedRunningTime="2026-09-09 17:21:52.135705643 +0000 UTC m=+306.484067116" Sep 09 17:21:53.109198 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:53.109163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" event={"ID":"8b58e555-f3b9-42cc-9e26-e822205efaf0","Type":"ContainerStarted","Data":"f4447517e3ebbe068c0c098bc7816e7d0a16fd3a2525635120ca1bbeb74d3631"} Sep 09 17:21:53.109531 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:53.109327 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:21:53.130904 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:53.130860 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" podStartSLOduration=2.717089775 podStartE2EDuration="6.130846917s" totalImagesPullingTime="3.413757142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:21:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:21:53.128332502 +0000 UTC m=+307.476693974" watchObservedRunningTime="2026-09-09 17:21:53.130846917 +0000 UTC m=+307.479208390" Sep 09 17:21:59.010051 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:59.010022 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:21:59.119257 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:21:59.119230 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c85b85d77-lpq9q" Sep 09 17:22:00.669174 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:00.669144 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:22:00.669602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:00.669197 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:22:00.673591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:00.673571 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:22:01.148676 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:01.148650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:22:09.908926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:09.908894 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-565b7696cc-fptf6"] Sep 09 17:22:15.330695 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.330661 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84dc44b9d6-wg8np" podUID="defe28f2-e89b-4223-ac1b-03f33831140e" containerName="console" containerID="cri-o://28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7" gracePeriod=15 Sep 09 17:22:15.598463 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.598410 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:22:15.698205 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698178 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-console-config\") pod \"defe28f2-e89b-4223-ac1b-03f33831140e\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " Sep 09 17:22:15.698336 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698225 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-serving-cert\") pod \"defe28f2-e89b-4223-ac1b-03f33831140e\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " Sep 09 17:22:15.698336 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698255 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-service-ca\") pod \"defe28f2-e89b-4223-ac1b-03f33831140e\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " Sep 09 17:22:15.698336 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698277 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swx27\" (UniqueName: \"kubernetes.io/projected/defe28f2-e89b-4223-ac1b-03f33831140e-kube-api-access-swx27\") pod \"defe28f2-e89b-4223-ac1b-03f33831140e\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " Sep 09 17:22:15.698336 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698296 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-oauth-serving-cert\") pod \"defe28f2-e89b-4223-ac1b-03f33831140e\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " Sep 09 17:22:15.698336 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698313 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-oauth-config\") pod \"defe28f2-e89b-4223-ac1b-03f33831140e\" (UID: \"defe28f2-e89b-4223-ac1b-03f33831140e\") " Sep 09 17:22:15.698647 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698616 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-console-config" (OuterVolumeSpecName: "console-config") pod "defe28f2-e89b-4223-ac1b-03f33831140e" (UID: "defe28f2-e89b-4223-ac1b-03f33831140e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:22:15.698756 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698656 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-service-ca" (OuterVolumeSpecName: "service-ca") pod "defe28f2-e89b-4223-ac1b-03f33831140e" (UID: "defe28f2-e89b-4223-ac1b-03f33831140e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:22:15.698805 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.698753 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "defe28f2-e89b-4223-ac1b-03f33831140e" (UID: "defe28f2-e89b-4223-ac1b-03f33831140e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:22:15.700560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.700523 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defe28f2-e89b-4223-ac1b-03f33831140e-kube-api-access-swx27" (OuterVolumeSpecName: "kube-api-access-swx27") pod "defe28f2-e89b-4223-ac1b-03f33831140e" (UID: "defe28f2-e89b-4223-ac1b-03f33831140e"). InnerVolumeSpecName "kube-api-access-swx27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:22:15.700560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.700547 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "defe28f2-e89b-4223-ac1b-03f33831140e" (UID: "defe28f2-e89b-4223-ac1b-03f33831140e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:22:15.700676 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.700606 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "defe28f2-e89b-4223-ac1b-03f33831140e" (UID: "defe28f2-e89b-4223-ac1b-03f33831140e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:22:15.799143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.799122 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:15.799143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.799142 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:15.799268 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.799150 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swx27\" (UniqueName: \"kubernetes.io/projected/defe28f2-e89b-4223-ac1b-03f33831140e-kube-api-access-swx27\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:15.799268 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.799163 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-oauth-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:15.799268 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.799177 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/defe28f2-e89b-4223-ac1b-03f33831140e-console-oauth-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:15.799268 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:15.799185 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/defe28f2-e89b-4223-ac1b-03f33831140e-console-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:16.200412 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.200376 2580 generic.go:356] "Generic (PLEG): container finished" podID="defe28f2-e89b-4223-ac1b-03f33831140e" containerID="28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7" exitCode=0 Sep 09 17:22:16.200570 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.200428 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84dc44b9d6-wg8np" Sep 09 17:22:16.200570 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.200457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dc44b9d6-wg8np" event={"ID":"defe28f2-e89b-4223-ac1b-03f33831140e","Type":"ContainerDied","Data":"28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7"} Sep 09 17:22:16.200570 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.200506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84dc44b9d6-wg8np" event={"ID":"defe28f2-e89b-4223-ac1b-03f33831140e","Type":"ContainerDied","Data":"53a2cc8c465c84428caa64c4662163964a2dcf031cd59cd7a109679af60ab63e"} Sep 09 17:22:16.200570 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.200526 2580 scope.go:117] "RemoveContainer" containerID="28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7" Sep 09 17:22:16.208308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.208292 2580 scope.go:117] "RemoveContainer" containerID="28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7" Sep 09 17:22:16.208576 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:22:16.208557 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7\": container with ID starting with 28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7 not found: ID does not exist" containerID="28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7" Sep 09 17:22:16.208627 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.208584 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7"} err="failed to get container status \"28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7\": rpc error: code = NotFound desc = could not find container \"28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7\": container with ID starting with 28239c5a8d31c41e313a9cf25652f894da331e028a000fe0cc61e908bd41aef7 not found: ID does not exist" Sep 09 17:22:16.215447 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.215427 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84dc44b9d6-wg8np"] Sep 09 17:22:16.219112 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:16.219095 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84dc44b9d6-wg8np"] Sep 09 17:22:18.129653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:18.129622 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="defe28f2-e89b-4223-ac1b-03f33831140e" path="/var/lib/kubelet/pods/defe28f2-e89b-4223-ac1b-03f33831140e/volumes" Sep 09 17:22:18.210774 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:18.210742 2580 generic.go:356] "Generic (PLEG): container finished" podID="e8028556-104d-4b1d-9914-138bf2e3f97b" containerID="5158251381c0b2a6b601f30616a3ff988eca806c1a568ee767315ea2ee7bcd34" exitCode=0 Sep 09 17:22:18.210913 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:18.210797 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" event={"ID":"e8028556-104d-4b1d-9914-138bf2e3f97b","Type":"ContainerDied","Data":"5158251381c0b2a6b601f30616a3ff988eca806c1a568ee767315ea2ee7bcd34"} Sep 09 17:22:18.211102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:18.211087 2580 scope.go:117] "RemoveContainer" containerID="5158251381c0b2a6b601f30616a3ff988eca806c1a568ee767315ea2ee7bcd34" Sep 09 17:22:19.220414 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:19.220379 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-4kj5h" event={"ID":"e8028556-104d-4b1d-9914-138bf2e3f97b","Type":"ContainerStarted","Data":"321037965fb6e0fd2c39a49c22770baa65fc97355e7fd2bb737112481dcc8d55"} Sep 09 17:22:23.237684 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:23.237652 2580 generic.go:356] "Generic (PLEG): container finished" podID="d11da5db-6654-463b-8880-b7aade5a657f" containerID="5a649647c09d474528363d88a2cc57c8363f32915cbb5d0f2c9f50b2e5fa0acd" exitCode=0 Sep 09 17:22:23.238121 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:23.237736 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" event={"ID":"d11da5db-6654-463b-8880-b7aade5a657f","Type":"ContainerDied","Data":"5a649647c09d474528363d88a2cc57c8363f32915cbb5d0f2c9f50b2e5fa0acd"} Sep 09 17:22:23.238121 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:23.238042 2580 scope.go:117] "RemoveContainer" containerID="5a649647c09d474528363d88a2cc57c8363f32915cbb5d0f2c9f50b2e5fa0acd" Sep 09 17:22:24.243517 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:24.243476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8sl5" event={"ID":"d11da5db-6654-463b-8880-b7aade5a657f","Type":"ContainerStarted","Data":"9c149fc0cd95a2c544ef47b0ce1ec37f7dda909697cea898dce89a2a283cc011"} Sep 09 17:22:26.927866 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:26.927838 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-565b7696cc-fptf6_daee458d-204d-4371-898f-5094dfe79c09/registry/0.log" Sep 09 17:22:27.499227 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:27.499197 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vcxxj_44fac9f1-d2e6-4397-bb60-47cf60343669/node-ca/0.log" Sep 09 17:22:28.101949 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:28.101924 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2nh4_0376df71-4ec7-4127-94d1-42318c23a135/serve-healthcheck-canary/0.log" Sep 09 17:22:34.928669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:34.928629 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" podUID="daee458d-204d-4371-898f-5094dfe79c09" containerName="registry" containerID="cri-o://ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c" gracePeriod=30 Sep 09 17:22:35.168773 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.168751 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:22:35.262321 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262298 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-bound-sa-token\") pod \"daee458d-204d-4371-898f-5094dfe79c09\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " Sep 09 17:22:35.262457 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-installation-pull-secrets\") pod \"daee458d-204d-4371-898f-5094dfe79c09\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " Sep 09 17:22:35.262457 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262384 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-trusted-ca\") pod \"daee458d-204d-4371-898f-5094dfe79c09\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " Sep 09 17:22:35.262457 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262417 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-image-registry-private-configuration\") pod \"daee458d-204d-4371-898f-5094dfe79c09\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " Sep 09 17:22:35.262457 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262442 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b76d9\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-kube-api-access-b76d9\") pod \"daee458d-204d-4371-898f-5094dfe79c09\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " Sep 09 17:22:35.262659 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262486 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/daee458d-204d-4371-898f-5094dfe79c09-ca-trust-extracted\") pod \"daee458d-204d-4371-898f-5094dfe79c09\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " Sep 09 17:22:35.262659 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262600 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") pod \"daee458d-204d-4371-898f-5094dfe79c09\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " Sep 09 17:22:35.262839 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262677 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-registry-certificates\") pod \"daee458d-204d-4371-898f-5094dfe79c09\" (UID: \"daee458d-204d-4371-898f-5094dfe79c09\") " Sep 09 17:22:35.262911 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.262888 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "daee458d-204d-4371-898f-5094dfe79c09" (UID: "daee458d-204d-4371-898f-5094dfe79c09"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:22:35.263380 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.263347 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "daee458d-204d-4371-898f-5094dfe79c09" (UID: "daee458d-204d-4371-898f-5094dfe79c09"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:22:35.265006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.264971 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-kube-api-access-b76d9" (OuterVolumeSpecName: "kube-api-access-b76d9") pod "daee458d-204d-4371-898f-5094dfe79c09" (UID: "daee458d-204d-4371-898f-5094dfe79c09"). InnerVolumeSpecName "kube-api-access-b76d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:22:35.265125 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.265096 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "daee458d-204d-4371-898f-5094dfe79c09" (UID: "daee458d-204d-4371-898f-5094dfe79c09"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:22:35.265186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.265150 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "daee458d-204d-4371-898f-5094dfe79c09" (UID: "daee458d-204d-4371-898f-5094dfe79c09"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:22:35.265186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.265170 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "daee458d-204d-4371-898f-5094dfe79c09" (UID: "daee458d-204d-4371-898f-5094dfe79c09"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:22:35.265254 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.265213 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "daee458d-204d-4371-898f-5094dfe79c09" (UID: "daee458d-204d-4371-898f-5094dfe79c09"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:22:35.271102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.271080 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daee458d-204d-4371-898f-5094dfe79c09-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "daee458d-204d-4371-898f-5094dfe79c09" (UID: "daee458d-204d-4371-898f-5094dfe79c09"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:22:35.290807 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.290778 2580 generic.go:356] "Generic (PLEG): container finished" podID="daee458d-204d-4371-898f-5094dfe79c09" containerID="ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c" exitCode=0 Sep 09 17:22:35.290918 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.290824 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" Sep 09 17:22:35.290918 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.290862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" event={"ID":"daee458d-204d-4371-898f-5094dfe79c09","Type":"ContainerDied","Data":"ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c"} Sep 09 17:22:35.290918 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.290914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565b7696cc-fptf6" event={"ID":"daee458d-204d-4371-898f-5094dfe79c09","Type":"ContainerDied","Data":"bd4917f71e015469ff669471594bb15104d377947b4c96209e9dbc691ec43771"} Sep 09 17:22:35.291024 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.290935 2580 scope.go:117] "RemoveContainer" containerID="ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c" Sep 09 17:22:35.300333 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.300319 2580 scope.go:117] "RemoveContainer" containerID="ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c" Sep 09 17:22:35.300551 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:22:35.300535 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c\": container with ID starting with ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c not found: ID does not exist" containerID="ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c" Sep 09 17:22:35.300593 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.300557 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c"} err="failed to get container status \"ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c\": rpc error: code = NotFound desc = could not find container \"ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c\": container with ID starting with ba73c985d78d1ff64d8ca521c65bd8ec4a21483a3d1f84f14636c241dda3866c not found: ID does not exist" Sep 09 17:22:35.311893 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.311870 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-565b7696cc-fptf6"] Sep 09 17:22:35.313745 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.313726 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-565b7696cc-fptf6"] Sep 09 17:22:35.363835 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.363814 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-bound-sa-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:35.363835 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.363835 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-installation-pull-secrets\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:35.363951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.363845 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-trusted-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:35.363951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.363854 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/daee458d-204d-4371-898f-5094dfe79c09-image-registry-private-configuration\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:35.363951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.363863 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b76d9\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-kube-api-access-b76d9\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:35.363951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.363872 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/daee458d-204d-4371-898f-5094dfe79c09-ca-trust-extracted\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:35.363951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.363880 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/daee458d-204d-4371-898f-5094dfe79c09-registry-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:35.363951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:35.363888 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/daee458d-204d-4371-898f-5094dfe79c09-registry-certificates\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:22:36.130899 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:36.130865 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daee458d-204d-4371-898f-5094dfe79c09" path="/var/lib/kubelet/pods/daee458d-204d-4371-898f-5094dfe79c09/volumes" Sep 09 17:22:59.060533 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:59.060493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:22:59.062867 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:59.062835 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9649a0b-cc7d-4eb0-9952-865e10002bd7-metrics-certs\") pod \"network-metrics-daemon-kkvpl\" (UID: \"f9649a0b-cc7d-4eb0-9952-865e10002bd7\") " pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:22:59.128015 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:59.127983 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kwqx\"" Sep 09 17:22:59.136155 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:59.136124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kkvpl" Sep 09 17:22:59.405102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:22:59.405021 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kkvpl"] Sep 09 17:22:59.407359 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:22:59.407335 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9649a0b_cc7d_4eb0_9952_865e10002bd7.slice/crio-b4d249bd371d1ef6be91e038869286d1f71743d71faa24ac732f536d3f586760 WatchSource:0}: Error finding container b4d249bd371d1ef6be91e038869286d1f71743d71faa24ac732f536d3f586760: Status 404 returned error can't find the container with id b4d249bd371d1ef6be91e038869286d1f71743d71faa24ac732f536d3f586760 Sep 09 17:23:00.399686 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:00.399652 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkvpl" event={"ID":"f9649a0b-cc7d-4eb0-9952-865e10002bd7","Type":"ContainerStarted","Data":"b4d249bd371d1ef6be91e038869286d1f71743d71faa24ac732f536d3f586760"} Sep 09 17:23:01.405802 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:01.405770 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkvpl" event={"ID":"f9649a0b-cc7d-4eb0-9952-865e10002bd7","Type":"ContainerStarted","Data":"d1bbdd26bf0eedd60ccec5064058e07ae92ef489fad4d91a8a04232615a1e2b1"} Sep 09 17:23:01.406196 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:01.405828 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kkvpl" event={"ID":"f9649a0b-cc7d-4eb0-9952-865e10002bd7","Type":"ContainerStarted","Data":"2d28a27818ae7d4c690fea86e8c03019a71a7b66e97298dc8d642e6b6430c1e8"} Sep 09 17:23:01.422696 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:01.422644 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kkvpl" podStartSLOduration=374.223085456 podStartE2EDuration="6m15.422628312s" totalImagesPullingTime="1.199542856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:23:01.421884248 +0000 UTC m=+375.770245720" watchObservedRunningTime="2026-09-09 17:23:01.422628312 +0000 UTC m=+375.770989785" Sep 09 17:23:06.368467 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.368432 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 17:23:06.369033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.368881 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daee458d-204d-4371-898f-5094dfe79c09" containerName="registry" Sep 09 17:23:06.369033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.368894 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="daee458d-204d-4371-898f-5094dfe79c09" containerName="registry" Sep 09 17:23:06.369033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.368904 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="defe28f2-e89b-4223-ac1b-03f33831140e" containerName="console" Sep 09 17:23:06.369033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.368909 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="defe28f2-e89b-4223-ac1b-03f33831140e" containerName="console" Sep 09 17:23:06.369033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.368959 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="daee458d-204d-4371-898f-5094dfe79c09" containerName="registry" Sep 09 17:23:06.369033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.368966 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="defe28f2-e89b-4223-ac1b-03f33831140e" containerName="console" Sep 09 17:23:06.372497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.372482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.375244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375219 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 17:23:06.375375 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375219 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 17:23:06.375375 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375248 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 17:23:06.375375 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375223 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 17:23:06.375375 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375298 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 17:23:06.375375 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375247 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 17:23:06.375584 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375521 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7pvbm\"" Sep 09 17:23:06.375780 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375763 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 17:23:06.375852 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.375771 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 17:23:06.380580 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.380560 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 17:23:06.383889 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.383869 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 17:23:06.522134 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522134 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522133 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-config-out\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522229 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522259 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nc8\" (UniqueName: \"kubernetes.io/projected/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-kube-api-access-w6nc8\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-web-config\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522369 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.522553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.522466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.623787 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.623668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.623787 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.623733 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.623787 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.623763 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-config-out\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.623787 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.623785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.623815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.623845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.623875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nc8\" (UniqueName: \"kubernetes.io/projected/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-kube-api-access-w6nc8\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.624016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-web-config\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.624082 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.624140 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.624226 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.624264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.624303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.624469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.624331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.625026 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.624757 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.626806 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.626784 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.627315 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.627053 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-web-config\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.627315 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.627266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.627463 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.627408 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.627521 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.627490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.627580 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.627564 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.627705 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.627682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.627793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.627682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-config-out\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.627866 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.627848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.628624 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.628601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.631632 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.631604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nc8\" (UniqueName: \"kubernetes.io/projected/3ecdd432-bb98-41f5-b6f8-e603f3eb601a-kube-api-access-w6nc8\") pod \"alertmanager-main-0\" (UID: \"3ecdd432-bb98-41f5-b6f8-e603f3eb601a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.682066 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.682015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 17:23:06.959969 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:06.959943 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 17:23:06.962702 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:23:06.962673 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecdd432_bb98_41f5_b6f8_e603f3eb601a.slice/crio-9685f740aaabbcb6de71d75c0d1156011ffa421aab653edb9db9abd8e7a778b9 WatchSource:0}: Error finding container 9685f740aaabbcb6de71d75c0d1156011ffa421aab653edb9db9abd8e7a778b9: Status 404 returned error can't find the container with id 9685f740aaabbcb6de71d75c0d1156011ffa421aab653edb9db9abd8e7a778b9 Sep 09 17:23:07.430549 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:07.430513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecdd432-bb98-41f5-b6f8-e603f3eb601a","Type":"ContainerStarted","Data":"9685f740aaabbcb6de71d75c0d1156011ffa421aab653edb9db9abd8e7a778b9"} Sep 09 17:23:08.435230 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.435149 2580 generic.go:356] "Generic (PLEG): container finished" podID="3ecdd432-bb98-41f5-b6f8-e603f3eb601a" containerID="0de0e2bd87e04a174fa44c6096d11659f9e1e1fdc796453586f9081e644559e7" exitCode=0 Sep 09 17:23:08.435564 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.435237 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecdd432-bb98-41f5-b6f8-e603f3eb601a","Type":"ContainerDied","Data":"0de0e2bd87e04a174fa44c6096d11659f9e1e1fdc796453586f9081e644559e7"} Sep 09 17:23:08.653080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.653046 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d47778895-8gfx6"] Sep 09 17:23:08.656653 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.656630 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.659274 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.659256 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 17:23:08.659375 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.659305 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 17:23:08.659462 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.659447 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-57cw8\"" Sep 09 17:23:08.659566 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.659550 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 17:23:08.659635 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.659621 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 17:23:08.659684 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.659627 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 17:23:08.664049 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.664028 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 17:23:08.672954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.672932 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d47778895-8gfx6"] Sep 09 17:23:08.745250 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.745166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-secret-telemeter-client\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.745250 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.745213 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-serving-certs-ca-bundle\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.745250 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.745242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.745458 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.745288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-federate-client-tls\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.745458 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.745357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-telemeter-client-tls\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.745458 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.745379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfhq\" (UniqueName: \"kubernetes.io/projected/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-kube-api-access-jlfhq\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.745458 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.745398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.745458 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.745415 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-metrics-client-ca\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.846493 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.846448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-secret-telemeter-client\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.846639 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.846517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-serving-certs-ca-bundle\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.846639 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.846559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.846639 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.846583 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-federate-client-tls\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.846639 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.846629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-telemeter-client-tls\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.846965 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.846655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfhq\" (UniqueName: \"kubernetes.io/projected/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-kube-api-access-jlfhq\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.846965 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.846680 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.846965 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.846734 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-metrics-client-ca\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.847490 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.847464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-metrics-client-ca\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.847586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.847494 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.847947 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.847921 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-serving-certs-ca-bundle\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.849157 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.849131 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.849275 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.849260 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-federate-client-tls\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.849321 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.849281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-secret-telemeter-client\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.849358 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.849327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-telemeter-client-tls\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.855847 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.855827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfhq\" (UniqueName: \"kubernetes.io/projected/0badb01a-0d3b-4ecc-95f6-2f81e94219f6-kube-api-access-jlfhq\") pod \"telemeter-client-5d47778895-8gfx6\" (UID: \"0badb01a-0d3b-4ecc-95f6-2f81e94219f6\") " pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:08.965858 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:08.965817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" Sep 09 17:23:09.239051 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:09.239025 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d47778895-8gfx6"] Sep 09 17:23:09.241565 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:23:09.241536 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0badb01a_0d3b_4ecc_95f6_2f81e94219f6.slice/crio-9963354c938a796dd3e055a230ee93b049f2a066ffce3fab9b707a86d539956b WatchSource:0}: Error finding container 9963354c938a796dd3e055a230ee93b049f2a066ffce3fab9b707a86d539956b: Status 404 returned error can't find the container with id 9963354c938a796dd3e055a230ee93b049f2a066ffce3fab9b707a86d539956b Sep 09 17:23:09.440098 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:09.440007 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" event={"ID":"0badb01a-0d3b-4ecc-95f6-2f81e94219f6","Type":"ContainerStarted","Data":"9963354c938a796dd3e055a230ee93b049f2a066ffce3fab9b707a86d539956b"} Sep 09 17:23:11.458879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.458543 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecdd432-bb98-41f5-b6f8-e603f3eb601a","Type":"ContainerStarted","Data":"6aa3a0187a2783fb5c8b57eceb84f5a580c546066c61e6b5aec2c2f383c72e2f"} Sep 09 17:23:11.458879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.458639 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecdd432-bb98-41f5-b6f8-e603f3eb601a","Type":"ContainerStarted","Data":"e7594ceea17c6651d5afb84579672f4daeef68de229da93f851187ba6a7ce711"} Sep 09 17:23:11.458879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.458687 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecdd432-bb98-41f5-b6f8-e603f3eb601a","Type":"ContainerStarted","Data":"4fe216c4d285ad93476cd8fa2ef5a2cb3745c61f18fff9e3febf8462e9ece676"} Sep 09 17:23:11.458879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.458735 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecdd432-bb98-41f5-b6f8-e603f3eb601a","Type":"ContainerStarted","Data":"d24964a6bb4c7af6072ec9841124b6faac23b966a5dbd2120c678f2196fa4ae6"} Sep 09 17:23:11.458879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.458771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecdd432-bb98-41f5-b6f8-e603f3eb601a","Type":"ContainerStarted","Data":"3708420e5d917da22644d2dd7c2c5a6c4a0ecb143e55937490efcdefe462102b"} Sep 09 17:23:11.458879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.458811 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecdd432-bb98-41f5-b6f8-e603f3eb601a","Type":"ContainerStarted","Data":"25606fc879bf35e78d13e5d6fda9aad04184fc973bfd00335cd3deb947ab9130"} Sep 09 17:23:11.461044 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.461020 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" event={"ID":"0badb01a-0d3b-4ecc-95f6-2f81e94219f6","Type":"ContainerStarted","Data":"58cc634107aa410386f6f95c74ea7330a9e3930ffb9ade8e5839d934ac13d276"} Sep 09 17:23:11.461145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.461065 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" event={"ID":"0badb01a-0d3b-4ecc-95f6-2f81e94219f6","Type":"ContainerStarted","Data":"666fd3f0d341bab8e5209cb92b2da2b094aa09da7d1184c8927634ab3d76fbc8"} Sep 09 17:23:11.487240 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:11.487189 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.473492196 podStartE2EDuration="5.487170682s" totalImagesPullingTime="2.997693869s" totalInitContainerRuntime="15.984617ms" isStatefulPod=false podCreationTimestamp="2026-09-09 17:23:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:23:11.48604386 +0000 UTC m=+385.834405334" watchObservedRunningTime="2026-09-09 17:23:11.487170682 +0000 UTC m=+385.835532158" Sep 09 17:23:12.466743 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.466691 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" event={"ID":"0badb01a-0d3b-4ecc-95f6-2f81e94219f6","Type":"ContainerStarted","Data":"5106e35e239734548abd98d00da211b808bc95dd71174fce8f86220b5814ba01"} Sep 09 17:23:12.491077 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.491020 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d47778895-8gfx6" podStartSLOduration=2.415107157 podStartE2EDuration="4.49100149s" totalImagesPullingTime="2.075894333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:23:12.49055818 +0000 UTC m=+386.838919655" watchObservedRunningTime="2026-09-09 17:23:12.49100149 +0000 UTC m=+386.839362965" Sep 09 17:23:12.576411 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.576374 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f59bf4ccf-grtcd"] Sep 09 17:23:12.582436 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.582410 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.588458 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.588431 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f59bf4ccf-grtcd"] Sep 09 17:23:12.679794 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.679762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9l8\" (UniqueName: \"kubernetes.io/projected/3af1fd4b-4f15-4411-80bf-a901df286bda-kube-api-access-tm9l8\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.679984 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.679805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-serving-cert\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.679984 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.679821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-trusted-ca-bundle\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.679984 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.679842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-oauth-config\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.679984 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.679858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-console-config\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.680166 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.679978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-oauth-serving-cert\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.680166 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.680027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-service-ca\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.781353 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.781319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-oauth-serving-cert\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.781353 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.781359 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-service-ca\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.781601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.781389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9l8\" (UniqueName: \"kubernetes.io/projected/3af1fd4b-4f15-4411-80bf-a901df286bda-kube-api-access-tm9l8\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.781601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.781410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-serving-cert\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.781601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.781427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-trusted-ca-bundle\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.781601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.781445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-oauth-config\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.781601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.781464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-console-config\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.782252 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.782220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-service-ca\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.782252 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.782238 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-console-config\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.782417 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.782395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-oauth-serving-cert\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.782460 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.782395 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-trusted-ca-bundle\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.783954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.783929 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-serving-cert\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.784020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.783957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-oauth-config\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.788298 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.788281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9l8\" (UniqueName: \"kubernetes.io/projected/3af1fd4b-4f15-4411-80bf-a901df286bda-kube-api-access-tm9l8\") pod \"console-5f59bf4ccf-grtcd\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:12.893550 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:12.893515 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:13.162649 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:13.162459 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f59bf4ccf-grtcd"] Sep 09 17:23:13.165169 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:23:13.165144 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af1fd4b_4f15_4411_80bf_a901df286bda.slice/crio-9c8dea573c170003cdee023022b92ca902f0d9fbe635feda3a7fe329f718b977 WatchSource:0}: Error finding container 9c8dea573c170003cdee023022b92ca902f0d9fbe635feda3a7fe329f718b977: Status 404 returned error can't find the container with id 9c8dea573c170003cdee023022b92ca902f0d9fbe635feda3a7fe329f718b977 Sep 09 17:23:13.472845 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:13.472765 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f59bf4ccf-grtcd" event={"ID":"3af1fd4b-4f15-4411-80bf-a901df286bda","Type":"ContainerStarted","Data":"ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b"} Sep 09 17:23:13.472845 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:13.472815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f59bf4ccf-grtcd" event={"ID":"3af1fd4b-4f15-4411-80bf-a901df286bda","Type":"ContainerStarted","Data":"9c8dea573c170003cdee023022b92ca902f0d9fbe635feda3a7fe329f718b977"} Sep 09 17:23:13.492281 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:13.492227 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f59bf4ccf-grtcd" podStartSLOduration=1.492209761 podStartE2EDuration="1.492209761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:23:13.49120315 +0000 UTC m=+387.839564624" watchObservedRunningTime="2026-09-09 17:23:13.492209761 +0000 UTC m=+387.840571233" Sep 09 17:23:22.893759 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:22.893696 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:22.894239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:22.893795 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:22.898306 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:22.898281 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:23.516789 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:23.516760 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:23:23.558810 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:23.558777 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fb79c4584-r96kj"] Sep 09 17:23:48.584430 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.584371 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fb79c4584-r96kj" podUID="0abada0d-d939-41fc-b683-6054ef953868" containerName="console" containerID="cri-o://aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd" gracePeriod=15 Sep 09 17:23:48.820792 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.820771 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:23:48.893164 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-service-ca\") pod \"0abada0d-d939-41fc-b683-6054ef953868\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " Sep 09 17:23:48.893164 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893131 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-oauth-config\") pod \"0abada0d-d939-41fc-b683-6054ef953868\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " Sep 09 17:23:48.893346 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893169 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-console-config\") pod \"0abada0d-d939-41fc-b683-6054ef953868\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " Sep 09 17:23:48.893346 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893200 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btdh5\" (UniqueName: \"kubernetes.io/projected/0abada0d-d939-41fc-b683-6054ef953868-kube-api-access-btdh5\") pod \"0abada0d-d939-41fc-b683-6054ef953868\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " Sep 09 17:23:48.893346 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893233 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-serving-cert\") pod \"0abada0d-d939-41fc-b683-6054ef953868\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " Sep 09 17:23:48.893346 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893274 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-oauth-serving-cert\") pod \"0abada0d-d939-41fc-b683-6054ef953868\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " Sep 09 17:23:48.893346 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893320 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-trusted-ca-bundle\") pod \"0abada0d-d939-41fc-b683-6054ef953868\" (UID: \"0abada0d-d939-41fc-b683-6054ef953868\") " Sep 09 17:23:48.893686 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-console-config" (OuterVolumeSpecName: "console-config") pod "0abada0d-d939-41fc-b683-6054ef953868" (UID: "0abada0d-d939-41fc-b683-6054ef953868"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:23:48.893686 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893674 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-service-ca" (OuterVolumeSpecName: "service-ca") pod "0abada0d-d939-41fc-b683-6054ef953868" (UID: "0abada0d-d939-41fc-b683-6054ef953868"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:23:48.893830 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893690 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0abada0d-d939-41fc-b683-6054ef953868" (UID: "0abada0d-d939-41fc-b683-6054ef953868"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:23:48.893888 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.893837 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0abada0d-d939-41fc-b683-6054ef953868" (UID: "0abada0d-d939-41fc-b683-6054ef953868"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:23:48.895274 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.895250 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0abada0d-d939-41fc-b683-6054ef953868" (UID: "0abada0d-d939-41fc-b683-6054ef953868"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:23:48.895370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.895302 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abada0d-d939-41fc-b683-6054ef953868-kube-api-access-btdh5" (OuterVolumeSpecName: "kube-api-access-btdh5") pod "0abada0d-d939-41fc-b683-6054ef953868" (UID: "0abada0d-d939-41fc-b683-6054ef953868"). InnerVolumeSpecName "kube-api-access-btdh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:23:48.895370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.895318 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0abada0d-d939-41fc-b683-6054ef953868" (UID: "0abada0d-d939-41fc-b683-6054ef953868"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:23:48.993950 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.993929 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-oauth-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:23:48.993950 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.993950 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-trusted-ca-bundle\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:23:48.994080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.993959 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:23:48.994080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.993967 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-oauth-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:23:48.994080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.993976 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0abada0d-d939-41fc-b683-6054ef953868-console-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:23:48.994080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.993984 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdh5\" (UniqueName: \"kubernetes.io/projected/0abada0d-d939-41fc-b683-6054ef953868-kube-api-access-btdh5\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:23:48.994080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:48.993993 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0abada0d-d939-41fc-b683-6054ef953868-console-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:23:49.632862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.632827 2580 generic.go:356] "Generic (PLEG): container finished" podID="0abada0d-d939-41fc-b683-6054ef953868" containerID="aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd" exitCode=0 Sep 09 17:23:49.632862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.632866 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb79c4584-r96kj" event={"ID":"0abada0d-d939-41fc-b683-6054ef953868","Type":"ContainerDied","Data":"aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd"} Sep 09 17:23:49.633284 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.632870 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb79c4584-r96kj" Sep 09 17:23:49.633284 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.632904 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb79c4584-r96kj" event={"ID":"0abada0d-d939-41fc-b683-6054ef953868","Type":"ContainerDied","Data":"a7095fb9d32bb8a64f562b3b410d90b40bd90f5049e3f66f1f44e19014cef204"} Sep 09 17:23:49.633284 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.632923 2580 scope.go:117] "RemoveContainer" containerID="aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd" Sep 09 17:23:49.641127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.641110 2580 scope.go:117] "RemoveContainer" containerID="aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd" Sep 09 17:23:49.641370 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:23:49.641356 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd\": container with ID starting with aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd not found: ID does not exist" containerID="aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd" Sep 09 17:23:49.641407 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.641378 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd"} err="failed to get container status \"aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd\": rpc error: code = NotFound desc = could not find container \"aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd\": container with ID starting with aadf5f944b376308f0ec607bdb99dff48c4744264320abe272e33799450630bd not found: ID does not exist" Sep 09 17:23:49.654050 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.654024 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fb79c4584-r96kj"] Sep 09 17:23:49.655404 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:49.655386 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fb79c4584-r96kj"] Sep 09 17:23:50.129705 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:23:50.129670 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0abada0d-d939-41fc-b683-6054ef953868" path="/var/lib/kubelet/pods/0abada0d-d939-41fc-b683-6054ef953868/volumes" Sep 09 17:24:20.070850 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.070774 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75d68fd4f8-b6krv"] Sep 09 17:24:20.071312 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.071128 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0abada0d-d939-41fc-b683-6054ef953868" containerName="console" Sep 09 17:24:20.071312 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.071143 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0abada0d-d939-41fc-b683-6054ef953868" containerName="console" Sep 09 17:24:20.071312 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.071238 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0abada0d-d939-41fc-b683-6054ef953868" containerName="console" Sep 09 17:24:20.074644 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.074621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.083587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.083558 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d68fd4f8-b6krv"] Sep 09 17:24:20.129980 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.129949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-oauth-config\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.129980 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.129982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-serving-cert\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.130229 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.130101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-oauth-serving-cert\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.130229 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.130141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-service-ca\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.130229 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.130162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjcc\" (UniqueName: \"kubernetes.io/projected/84fcab89-ebed-4708-8b6c-64f9406ec1d3-kube-api-access-qqjcc\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.130380 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.130252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-trusted-ca-bundle\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.130380 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.130306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-config\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.231430 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.231390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-oauth-serving-cert\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.231430 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.231434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-service-ca\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.231688 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.231454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjcc\" (UniqueName: \"kubernetes.io/projected/84fcab89-ebed-4708-8b6c-64f9406ec1d3-kube-api-access-qqjcc\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.231688 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.231472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-trusted-ca-bundle\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.231688 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.231608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-config\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.231688 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.231684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-oauth-config\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.231917 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.231736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-serving-cert\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.232256 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.232226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-oauth-serving-cert\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.232414 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.232392 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-config\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.232514 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.232379 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-service-ca\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.232514 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.232451 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-trusted-ca-bundle\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.234185 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.234166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-serving-cert\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.234261 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.234204 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-oauth-config\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.239917 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.239899 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjcc\" (UniqueName: \"kubernetes.io/projected/84fcab89-ebed-4708-8b6c-64f9406ec1d3-kube-api-access-qqjcc\") pod \"console-75d68fd4f8-b6krv\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.384860 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.384766 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:20.653128 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.653052 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75d68fd4f8-b6krv"] Sep 09 17:24:20.655809 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:24:20.655777 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fcab89_ebed_4708_8b6c_64f9406ec1d3.slice/crio-c1db7f68e798148bf97f7ed8897ab16b07a10c10fa163e6e280ce17874a478ad WatchSource:0}: Error finding container c1db7f68e798148bf97f7ed8897ab16b07a10c10fa163e6e280ce17874a478ad: Status 404 returned error can't find the container with id c1db7f68e798148bf97f7ed8897ab16b07a10c10fa163e6e280ce17874a478ad Sep 09 17:24:20.765158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.765112 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d68fd4f8-b6krv" event={"ID":"84fcab89-ebed-4708-8b6c-64f9406ec1d3","Type":"ContainerStarted","Data":"538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64"} Sep 09 17:24:20.765158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.765164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d68fd4f8-b6krv" event={"ID":"84fcab89-ebed-4708-8b6c-64f9406ec1d3","Type":"ContainerStarted","Data":"c1db7f68e798148bf97f7ed8897ab16b07a10c10fa163e6e280ce17874a478ad"} Sep 09 17:24:20.781872 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:20.781823 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75d68fd4f8-b6krv" podStartSLOduration=0.781805822 podStartE2EDuration="781.805822ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:24:20.781281858 +0000 UTC m=+455.129643332" watchObservedRunningTime="2026-09-09 17:24:20.781805822 +0000 UTC m=+455.130167295" Sep 09 17:24:30.385804 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:30.385770 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:30.386163 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:30.385836 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:30.390356 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:30.390331 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:30.812334 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:30.812307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:24:30.865880 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:30.865849 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f59bf4ccf-grtcd"] Sep 09 17:24:55.886077 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:55.886035 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f59bf4ccf-grtcd" podUID="3af1fd4b-4f15-4411-80bf-a901df286bda" containerName="console" containerID="cri-o://ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b" gracePeriod=15 Sep 09 17:24:56.115855 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.115830 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:24:56.227304 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-console-config\") pod \"3af1fd4b-4f15-4411-80bf-a901df286bda\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " Sep 09 17:24:56.227304 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227291 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-trusted-ca-bundle\") pod \"3af1fd4b-4f15-4411-80bf-a901df286bda\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " Sep 09 17:24:56.227493 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227326 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-service-ca\") pod \"3af1fd4b-4f15-4411-80bf-a901df286bda\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " Sep 09 17:24:56.227493 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227346 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm9l8\" (UniqueName: \"kubernetes.io/projected/3af1fd4b-4f15-4411-80bf-a901df286bda-kube-api-access-tm9l8\") pod \"3af1fd4b-4f15-4411-80bf-a901df286bda\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " Sep 09 17:24:56.227493 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227389 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-oauth-config\") pod \"3af1fd4b-4f15-4411-80bf-a901df286bda\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " Sep 09 17:24:56.227493 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227407 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-oauth-serving-cert\") pod \"3af1fd4b-4f15-4411-80bf-a901df286bda\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " Sep 09 17:24:56.227493 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227462 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-serving-cert\") pod \"3af1fd4b-4f15-4411-80bf-a901df286bda\" (UID: \"3af1fd4b-4f15-4411-80bf-a901df286bda\") " Sep 09 17:24:56.227800 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227680 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-console-config" (OuterVolumeSpecName: "console-config") pod "3af1fd4b-4f15-4411-80bf-a901df286bda" (UID: "3af1fd4b-4f15-4411-80bf-a901df286bda"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:24:56.227800 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227752 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-service-ca" (OuterVolumeSpecName: "service-ca") pod "3af1fd4b-4f15-4411-80bf-a901df286bda" (UID: "3af1fd4b-4f15-4411-80bf-a901df286bda"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:24:56.227879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3af1fd4b-4f15-4411-80bf-a901df286bda" (UID: "3af1fd4b-4f15-4411-80bf-a901df286bda"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:24:56.227879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.227842 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3af1fd4b-4f15-4411-80bf-a901df286bda" (UID: "3af1fd4b-4f15-4411-80bf-a901df286bda"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:24:56.229513 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.229487 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3af1fd4b-4f15-4411-80bf-a901df286bda" (UID: "3af1fd4b-4f15-4411-80bf-a901df286bda"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:24:56.229603 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.229496 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3af1fd4b-4f15-4411-80bf-a901df286bda" (UID: "3af1fd4b-4f15-4411-80bf-a901df286bda"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:24:56.229603 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.229547 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af1fd4b-4f15-4411-80bf-a901df286bda-kube-api-access-tm9l8" (OuterVolumeSpecName: "kube-api-access-tm9l8") pod "3af1fd4b-4f15-4411-80bf-a901df286bda" (UID: "3af1fd4b-4f15-4411-80bf-a901df286bda"). InnerVolumeSpecName "kube-api-access-tm9l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:24:56.328622 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.328596 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-trusted-ca-bundle\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:24:56.328622 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.328617 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:24:56.328776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.328626 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm9l8\" (UniqueName: \"kubernetes.io/projected/3af1fd4b-4f15-4411-80bf-a901df286bda-kube-api-access-tm9l8\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:24:56.328776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.328635 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-oauth-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:24:56.328776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.328643 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-oauth-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:24:56.328776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.328651 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3af1fd4b-4f15-4411-80bf-a901df286bda-console-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:24:56.328776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.328659 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3af1fd4b-4f15-4411-80bf-a901df286bda-console-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:24:56.923944 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.923913 2580 generic.go:356] "Generic (PLEG): container finished" podID="3af1fd4b-4f15-4411-80bf-a901df286bda" containerID="ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b" exitCode=0 Sep 09 17:24:56.924358 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.923954 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f59bf4ccf-grtcd" Sep 09 17:24:56.924358 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.923997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f59bf4ccf-grtcd" event={"ID":"3af1fd4b-4f15-4411-80bf-a901df286bda","Type":"ContainerDied","Data":"ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b"} Sep 09 17:24:56.924358 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.924046 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f59bf4ccf-grtcd" event={"ID":"3af1fd4b-4f15-4411-80bf-a901df286bda","Type":"ContainerDied","Data":"9c8dea573c170003cdee023022b92ca902f0d9fbe635feda3a7fe329f718b977"} Sep 09 17:24:56.924358 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.924065 2580 scope.go:117] "RemoveContainer" containerID="ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b" Sep 09 17:24:56.932833 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.932818 2580 scope.go:117] "RemoveContainer" containerID="ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b" Sep 09 17:24:56.933090 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:24:56.933072 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b\": container with ID starting with ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b not found: ID does not exist" containerID="ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b" Sep 09 17:24:56.933124 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.933099 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b"} err="failed to get container status \"ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b\": rpc error: code = NotFound desc = could not find container \"ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b\": container with ID starting with ecfacc82172c1cb1b9b3461bb6cf82ec68cad2d471271bc09af019016817536b not found: ID does not exist" Sep 09 17:24:56.944605 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.944583 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f59bf4ccf-grtcd"] Sep 09 17:24:56.946696 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:56.946677 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f59bf4ccf-grtcd"] Sep 09 17:24:58.132882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:24:58.132850 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af1fd4b-4f15-4411-80bf-a901df286bda" path="/var/lib/kubelet/pods/3af1fd4b-4f15-4411-80bf-a901df286bda/volumes" Sep 09 17:26:02.666526 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.666492 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf"] Sep 09 17:26:02.667008 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.666843 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af1fd4b-4f15-4411-80bf-a901df286bda" containerName="console" Sep 09 17:26:02.667008 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.666854 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af1fd4b-4f15-4411-80bf-a901df286bda" containerName="console" Sep 09 17:26:02.667008 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.666915 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af1fd4b-4f15-4411-80bf-a901df286bda" containerName="console" Sep 09 17:26:02.670319 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.670303 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" Sep 09 17:26:02.672917 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.672891 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 09 17:26:02.672917 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.672911 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 09 17:26:02.673083 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.672952 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ktf2q\"" Sep 09 17:26:02.681887 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.681866 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf"] Sep 09 17:26:02.745986 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.745952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7309b66d-934a-4b3f-ad95-7ee83f8dcde6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c4nvf\" (UID: \"7309b66d-934a-4b3f-ad95-7ee83f8dcde6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" Sep 09 17:26:02.745986 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.745986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5fn5\" (UniqueName: \"kubernetes.io/projected/7309b66d-934a-4b3f-ad95-7ee83f8dcde6-kube-api-access-h5fn5\") pod \"cert-manager-operator-controller-manager-55c444bf86-c4nvf\" (UID: \"7309b66d-934a-4b3f-ad95-7ee83f8dcde6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" Sep 09 17:26:02.847271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.847235 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7309b66d-934a-4b3f-ad95-7ee83f8dcde6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c4nvf\" (UID: \"7309b66d-934a-4b3f-ad95-7ee83f8dcde6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" Sep 09 17:26:02.847271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.847274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5fn5\" (UniqueName: \"kubernetes.io/projected/7309b66d-934a-4b3f-ad95-7ee83f8dcde6-kube-api-access-h5fn5\") pod \"cert-manager-operator-controller-manager-55c444bf86-c4nvf\" (UID: \"7309b66d-934a-4b3f-ad95-7ee83f8dcde6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" Sep 09 17:26:02.847655 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.847637 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7309b66d-934a-4b3f-ad95-7ee83f8dcde6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c4nvf\" (UID: \"7309b66d-934a-4b3f-ad95-7ee83f8dcde6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" Sep 09 17:26:02.858498 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.858465 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5fn5\" (UniqueName: \"kubernetes.io/projected/7309b66d-934a-4b3f-ad95-7ee83f8dcde6-kube-api-access-h5fn5\") pod \"cert-manager-operator-controller-manager-55c444bf86-c4nvf\" (UID: \"7309b66d-934a-4b3f-ad95-7ee83f8dcde6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" Sep 09 17:26:02.979544 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:02.979459 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" Sep 09 17:26:03.259231 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:03.259202 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf"] Sep 09 17:26:03.262221 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:26:03.262196 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7309b66d_934a_4b3f_ad95_7ee83f8dcde6.slice/crio-e9940158d71377ad3a98eae5ade0965d47dbb1f523bd50180f6162e941737582 WatchSource:0}: Error finding container e9940158d71377ad3a98eae5ade0965d47dbb1f523bd50180f6162e941737582: Status 404 returned error can't find the container with id e9940158d71377ad3a98eae5ade0965d47dbb1f523bd50180f6162e941737582 Sep 09 17:26:04.204948 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:04.204912 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" event={"ID":"7309b66d-934a-4b3f-ad95-7ee83f8dcde6","Type":"ContainerStarted","Data":"e9940158d71377ad3a98eae5ade0965d47dbb1f523bd50180f6162e941737582"} Sep 09 17:26:06.215377 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:06.215286 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" event={"ID":"7309b66d-934a-4b3f-ad95-7ee83f8dcde6","Type":"ContainerStarted","Data":"c5fea0738683e8d60823f23a8b28ded26b2e0e7213b890a1330e053d3deb2488"} Sep 09 17:26:06.235992 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:06.235951 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c4nvf" podStartSLOduration=1.695974927 podStartE2EDuration="4.235936061s" totalImagesPullingTime="2.539961134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:26:06.233863984 +0000 UTC m=+560.582225455" watchObservedRunningTime="2026-09-09 17:26:06.235936061 +0000 UTC m=+560.584297534" Sep 09 17:26:09.380520 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.380485 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-h8x54"] Sep 09 17:26:09.383970 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.383954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" Sep 09 17:26:09.386657 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.386635 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 17:26:09.387830 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.387767 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-thvlj\"" Sep 09 17:26:09.387830 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.387813 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 17:26:09.389999 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.389979 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-h8x54"] Sep 09 17:26:09.499961 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.499934 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkj7t\" (UniqueName: \"kubernetes.io/projected/027b162f-cb73-40ae-a68c-fec4f436e6f4-kube-api-access-mkj7t\") pod \"cert-manager-cainjector-559cdd8c87-h8x54\" (UID: \"027b162f-cb73-40ae-a68c-fec4f436e6f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" Sep 09 17:26:09.500080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.499998 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/027b162f-cb73-40ae-a68c-fec4f436e6f4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-h8x54\" (UID: \"027b162f-cb73-40ae-a68c-fec4f436e6f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" Sep 09 17:26:09.600384 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.600358 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/027b162f-cb73-40ae-a68c-fec4f436e6f4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-h8x54\" (UID: \"027b162f-cb73-40ae-a68c-fec4f436e6f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" Sep 09 17:26:09.600479 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.600390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkj7t\" (UniqueName: \"kubernetes.io/projected/027b162f-cb73-40ae-a68c-fec4f436e6f4-kube-api-access-mkj7t\") pod \"cert-manager-cainjector-559cdd8c87-h8x54\" (UID: \"027b162f-cb73-40ae-a68c-fec4f436e6f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" Sep 09 17:26:09.608350 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.608321 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/027b162f-cb73-40ae-a68c-fec4f436e6f4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-h8x54\" (UID: \"027b162f-cb73-40ae-a68c-fec4f436e6f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" Sep 09 17:26:09.608606 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.608587 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkj7t\" (UniqueName: \"kubernetes.io/projected/027b162f-cb73-40ae-a68c-fec4f436e6f4-kube-api-access-mkj7t\") pod \"cert-manager-cainjector-559cdd8c87-h8x54\" (UID: \"027b162f-cb73-40ae-a68c-fec4f436e6f4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" Sep 09 17:26:09.703738 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.703661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" Sep 09 17:26:09.974139 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:09.974115 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-h8x54"] Sep 09 17:26:09.976049 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:26:09.976017 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod027b162f_cb73_40ae_a68c_fec4f436e6f4.slice/crio-0d6e8e02fda6f703777b843b4a0da069e0843d42f6e7ae3f78c06ee948c23747 WatchSource:0}: Error finding container 0d6e8e02fda6f703777b843b4a0da069e0843d42f6e7ae3f78c06ee948c23747: Status 404 returned error can't find the container with id 0d6e8e02fda6f703777b843b4a0da069e0843d42f6e7ae3f78c06ee948c23747 Sep 09 17:26:10.231537 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:10.231469 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" event={"ID":"027b162f-cb73-40ae-a68c-fec4f436e6f4","Type":"ContainerStarted","Data":"0d6e8e02fda6f703777b843b4a0da069e0843d42f6e7ae3f78c06ee948c23747"} Sep 09 17:26:13.173482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.173404 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-szbds"] Sep 09 17:26:13.176929 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.176899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:13.179465 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.179441 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-pdrlh\"" Sep 09 17:26:13.184737 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.184691 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-szbds"] Sep 09 17:26:13.249197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.249165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" event={"ID":"027b162f-cb73-40ae-a68c-fec4f436e6f4","Type":"ContainerStarted","Data":"0cb18331036f12869f65d83c369cf3dcf4dc3c1bd1a7cc1663147686d74cc8fa"} Sep 09 17:26:13.270613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.270556 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-h8x54" podStartSLOduration=1.431837535 podStartE2EDuration="4.270536244s" totalImagesPullingTime="2.838698709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:26:13.26957963 +0000 UTC m=+567.617941104" watchObservedRunningTime="2026-09-09 17:26:13.270536244 +0000 UTC m=+567.618897717" Sep 09 17:26:13.332502 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.332474 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4752ed94-5de4-4d9b-8c21-edf74e50db4c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-szbds\" (UID: \"4752ed94-5de4-4d9b-8c21-edf74e50db4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:13.332634 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.332530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvqt5\" (UniqueName: \"kubernetes.io/projected/4752ed94-5de4-4d9b-8c21-edf74e50db4c-kube-api-access-rvqt5\") pod \"cert-manager-webhook-6486ffd997-szbds\" (UID: \"4752ed94-5de4-4d9b-8c21-edf74e50db4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:13.433784 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.433694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4752ed94-5de4-4d9b-8c21-edf74e50db4c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-szbds\" (UID: \"4752ed94-5de4-4d9b-8c21-edf74e50db4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:13.433905 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.433824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvqt5\" (UniqueName: \"kubernetes.io/projected/4752ed94-5de4-4d9b-8c21-edf74e50db4c-kube-api-access-rvqt5\") pod \"cert-manager-webhook-6486ffd997-szbds\" (UID: \"4752ed94-5de4-4d9b-8c21-edf74e50db4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:13.441224 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.441194 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4752ed94-5de4-4d9b-8c21-edf74e50db4c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-szbds\" (UID: \"4752ed94-5de4-4d9b-8c21-edf74e50db4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:13.441327 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.441311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvqt5\" (UniqueName: \"kubernetes.io/projected/4752ed94-5de4-4d9b-8c21-edf74e50db4c-kube-api-access-rvqt5\") pod \"cert-manager-webhook-6486ffd997-szbds\" (UID: \"4752ed94-5de4-4d9b-8c21-edf74e50db4c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:13.487196 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.487162 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:13.756519 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:13.756473 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-szbds"] Sep 09 17:26:13.760294 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:26:13.760264 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4752ed94_5de4_4d9b_8c21_edf74e50db4c.slice/crio-163454f7479183611a0461b19d2646d32b117e02584e610c117fc8f29c031625 WatchSource:0}: Error finding container 163454f7479183611a0461b19d2646d32b117e02584e610c117fc8f29c031625: Status 404 returned error can't find the container with id 163454f7479183611a0461b19d2646d32b117e02584e610c117fc8f29c031625 Sep 09 17:26:14.255256 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:14.255223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" event={"ID":"4752ed94-5de4-4d9b-8c21-edf74e50db4c","Type":"ContainerStarted","Data":"7071eb6592519ea06c911e266be31408e7ec195a847ded206f0bb0a683fa956e"} Sep 09 17:26:14.255621 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:14.255268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" event={"ID":"4752ed94-5de4-4d9b-8c21-edf74e50db4c","Type":"ContainerStarted","Data":"163454f7479183611a0461b19d2646d32b117e02584e610c117fc8f29c031625"} Sep 09 17:26:14.255621 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:14.255314 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:14.271313 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:14.271201 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" podStartSLOduration=1.271185466 podStartE2EDuration="1.271185466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:26:14.271012325 +0000 UTC m=+568.619373799" watchObservedRunningTime="2026-09-09 17:26:14.271185466 +0000 UTC m=+568.619546940" Sep 09 17:26:19.239286 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.239251 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jrjss"] Sep 09 17:26:19.242671 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.242653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jrjss" Sep 09 17:26:19.245080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.245063 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lmjzw\"" Sep 09 17:26:19.251572 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.251547 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jrjss"] Sep 09 17:26:19.281120 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.281087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af4984d4-3bea-499a-89a4-e238504e288e-bound-sa-token\") pod \"cert-manager-55b4d954d5-jrjss\" (UID: \"af4984d4-3bea-499a-89a4-e238504e288e\") " pod="cert-manager/cert-manager-55b4d954d5-jrjss" Sep 09 17:26:19.281223 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.281154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddkq\" (UniqueName: \"kubernetes.io/projected/af4984d4-3bea-499a-89a4-e238504e288e-kube-api-access-bddkq\") pod \"cert-manager-55b4d954d5-jrjss\" (UID: \"af4984d4-3bea-499a-89a4-e238504e288e\") " pod="cert-manager/cert-manager-55b4d954d5-jrjss" Sep 09 17:26:19.382332 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.382309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af4984d4-3bea-499a-89a4-e238504e288e-bound-sa-token\") pod \"cert-manager-55b4d954d5-jrjss\" (UID: \"af4984d4-3bea-499a-89a4-e238504e288e\") " pod="cert-manager/cert-manager-55b4d954d5-jrjss" Sep 09 17:26:19.382432 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.382356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddkq\" (UniqueName: \"kubernetes.io/projected/af4984d4-3bea-499a-89a4-e238504e288e-kube-api-access-bddkq\") pod \"cert-manager-55b4d954d5-jrjss\" (UID: \"af4984d4-3bea-499a-89a4-e238504e288e\") " pod="cert-manager/cert-manager-55b4d954d5-jrjss" Sep 09 17:26:19.390669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.390642 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/af4984d4-3bea-499a-89a4-e238504e288e-bound-sa-token\") pod \"cert-manager-55b4d954d5-jrjss\" (UID: \"af4984d4-3bea-499a-89a4-e238504e288e\") " pod="cert-manager/cert-manager-55b4d954d5-jrjss" Sep 09 17:26:19.390838 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.390783 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddkq\" (UniqueName: \"kubernetes.io/projected/af4984d4-3bea-499a-89a4-e238504e288e-kube-api-access-bddkq\") pod \"cert-manager-55b4d954d5-jrjss\" (UID: \"af4984d4-3bea-499a-89a4-e238504e288e\") " pod="cert-manager/cert-manager-55b4d954d5-jrjss" Sep 09 17:26:19.552880 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.552854 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jrjss" Sep 09 17:26:19.820851 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:19.820783 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jrjss"] Sep 09 17:26:19.824029 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:26:19.824004 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf4984d4_3bea_499a_89a4_e238504e288e.slice/crio-4444be5102cafeb87ad2be91d3a8ab5ea88c4b71dae3abe7434f7d2ab0354de6 WatchSource:0}: Error finding container 4444be5102cafeb87ad2be91d3a8ab5ea88c4b71dae3abe7434f7d2ab0354de6: Status 404 returned error can't find the container with id 4444be5102cafeb87ad2be91d3a8ab5ea88c4b71dae3abe7434f7d2ab0354de6 Sep 09 17:26:20.261364 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:20.261337 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-szbds" Sep 09 17:26:20.282014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:20.281977 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jrjss" event={"ID":"af4984d4-3bea-499a-89a4-e238504e288e","Type":"ContainerStarted","Data":"906a5c680ce89f93bf665649084445f01004e1820fbfd545738c6191373bb896"} Sep 09 17:26:20.282152 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:20.282017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jrjss" event={"ID":"af4984d4-3bea-499a-89a4-e238504e288e","Type":"ContainerStarted","Data":"4444be5102cafeb87ad2be91d3a8ab5ea88c4b71dae3abe7434f7d2ab0354de6"} Sep 09 17:26:20.301636 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:20.301587 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jrjss" podStartSLOduration=1.301571461 podStartE2EDuration="1.301571461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:26:20.300164724 +0000 UTC m=+574.648526200" watchObservedRunningTime="2026-09-09 17:26:20.301571461 +0000 UTC m=+574.649932933" Sep 09 17:26:46.168518 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:46.168425 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:26:46.172168 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:46.171562 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:26:46.172168 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:46.171932 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:26:46.174985 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:46.174968 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:26:48.965837 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:48.965792 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d68fd4f8-b6krv"] Sep 09 17:26:57.402261 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.402226 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n"] Sep 09 17:26:57.406242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.406215 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.409147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.409120 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tsgcn\"" Sep 09 17:26:57.409323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.409157 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 17:26:57.409323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.409168 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 17:26:57.409323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.409178 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 17:26:57.409471 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.409458 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 17:26:57.420370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.420347 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n"] Sep 09 17:26:57.480440 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.480404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8qv\" (UniqueName: \"kubernetes.io/projected/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-kube-api-access-bw8qv\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.480440 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.480440 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-manifests\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.480645 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.480463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.480645 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.480516 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-tmp\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.480645 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.480596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-webhook-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.581638 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.581605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-webhook-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.581843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.581668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8qv\" (UniqueName: \"kubernetes.io/projected/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-kube-api-access-bw8qv\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.581843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.581689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-manifests\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.581843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.581732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.581843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.581757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-tmp\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.582159 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.582137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-manifests\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.584101 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.584072 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-tmp\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.584211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.584189 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.584211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.584200 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-webhook-cert\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.590469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.590450 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8qv\" (UniqueName: \"kubernetes.io/projected/0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c-kube-api-access-bw8qv\") pod \"opendatahub-operator-controller-manager-cdfc9465f-zk66n\" (UID: \"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c\") " pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:57.717143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:57.717066 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:26:58.021588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:58.021561 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n"] Sep 09 17:26:58.022088 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:26:58.022065 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3a6fab_25ff_4c1f_ac31_0bc30fe1473c.slice/crio-03ce0ffbe90fa8cc598c2ca33de4f107a7baeb5ca695516a6f848baa85364482 WatchSource:0}: Error finding container 03ce0ffbe90fa8cc598c2ca33de4f107a7baeb5ca695516a6f848baa85364482: Status 404 returned error can't find the container with id 03ce0ffbe90fa8cc598c2ca33de4f107a7baeb5ca695516a6f848baa85364482 Sep 09 17:26:58.023759 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:58.023740 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:26:58.459981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:26:58.459899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" event={"ID":"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c","Type":"ContainerStarted","Data":"03ce0ffbe90fa8cc598c2ca33de4f107a7baeb5ca695516a6f848baa85364482"} Sep 09 17:27:01.477597 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:01.477513 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" event={"ID":"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c","Type":"ContainerStarted","Data":"a7404fd10e0d9aef3825aa56571d1b310ec24ca487ed29f5d55dadc3e72ecd17"} Sep 09 17:27:02.484224 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:02.484135 2580 generic.go:356] "Generic (PLEG): container finished" podID="0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c" containerID="a7404fd10e0d9aef3825aa56571d1b310ec24ca487ed29f5d55dadc3e72ecd17" exitCode=0 Sep 09 17:27:02.484224 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:02.484213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" event={"ID":"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c","Type":"ContainerDied","Data":"a7404fd10e0d9aef3825aa56571d1b310ec24ca487ed29f5d55dadc3e72ecd17"} Sep 09 17:27:03.490908 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:03.490874 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" event={"ID":"0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c","Type":"ContainerStarted","Data":"523030b03551227b80e5d44b0b4c2adc2dd1a15ff7c64eaa412b9265a7e58763"} Sep 09 17:27:03.491294 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:03.490969 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:27:03.514981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:03.514928 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" podStartSLOduration=2.052521337 podStartE2EDuration="6.514913002s" totalImagesPullingTime="3.446673605s" totalInitContainerRuntime="1.01571806s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:26:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:27:03.512475592 +0000 UTC m=+617.860837077" watchObservedRunningTime="2026-09-09 17:27:03.514913002 +0000 UTC m=+617.863274475" Sep 09 17:27:13.985793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:13.985655 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75d68fd4f8-b6krv" podUID="84fcab89-ebed-4708-8b6c-64f9406ec1d3" containerName="console" containerID="cri-o://538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64" gracePeriod=15 Sep 09 17:27:14.222886 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.222863 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:27:14.327820 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.327784 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqjcc\" (UniqueName: \"kubernetes.io/projected/84fcab89-ebed-4708-8b6c-64f9406ec1d3-kube-api-access-qqjcc\") pod \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " Sep 09 17:27:14.328012 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.327832 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-trusted-ca-bundle\") pod \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " Sep 09 17:27:14.328012 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.327867 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-oauth-serving-cert\") pod \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " Sep 09 17:27:14.328012 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.327887 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-oauth-config\") pod \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " Sep 09 17:27:14.328012 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.327948 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-config\") pod \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " Sep 09 17:27:14.328012 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.328006 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-serving-cert\") pod \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " Sep 09 17:27:14.328281 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.328037 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-service-ca\") pod \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\" (UID: \"84fcab89-ebed-4708-8b6c-64f9406ec1d3\") " Sep 09 17:27:14.328370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.328341 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "84fcab89-ebed-4708-8b6c-64f9406ec1d3" (UID: "84fcab89-ebed-4708-8b6c-64f9406ec1d3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:27:14.328454 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.328366 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "84fcab89-ebed-4708-8b6c-64f9406ec1d3" (UID: "84fcab89-ebed-4708-8b6c-64f9406ec1d3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:27:14.328508 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.328462 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-config" (OuterVolumeSpecName: "console-config") pod "84fcab89-ebed-4708-8b6c-64f9406ec1d3" (UID: "84fcab89-ebed-4708-8b6c-64f9406ec1d3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:27:14.328549 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.328516 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-service-ca" (OuterVolumeSpecName: "service-ca") pod "84fcab89-ebed-4708-8b6c-64f9406ec1d3" (UID: "84fcab89-ebed-4708-8b6c-64f9406ec1d3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:27:14.330235 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.330214 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fcab89-ebed-4708-8b6c-64f9406ec1d3-kube-api-access-qqjcc" (OuterVolumeSpecName: "kube-api-access-qqjcc") pod "84fcab89-ebed-4708-8b6c-64f9406ec1d3" (UID: "84fcab89-ebed-4708-8b6c-64f9406ec1d3"). InnerVolumeSpecName "kube-api-access-qqjcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:27:14.330490 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.330471 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "84fcab89-ebed-4708-8b6c-64f9406ec1d3" (UID: "84fcab89-ebed-4708-8b6c-64f9406ec1d3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:27:14.330585 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.330497 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "84fcab89-ebed-4708-8b6c-64f9406ec1d3" (UID: "84fcab89-ebed-4708-8b6c-64f9406ec1d3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:27:14.428849 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.428814 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:14.428849 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.428845 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:14.428849 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.428854 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:14.429082 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.428863 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjcc\" (UniqueName: \"kubernetes.io/projected/84fcab89-ebed-4708-8b6c-64f9406ec1d3-kube-api-access-qqjcc\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:14.429082 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.428871 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-trusted-ca-bundle\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:14.429082 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.428880 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/84fcab89-ebed-4708-8b6c-64f9406ec1d3-oauth-serving-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:14.429082 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.428888 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/84fcab89-ebed-4708-8b6c-64f9406ec1d3-console-oauth-config\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:14.497707 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.497681 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cdfc9465f-zk66n" Sep 09 17:27:14.545732 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.545690 2580 generic.go:356] "Generic (PLEG): container finished" podID="84fcab89-ebed-4708-8b6c-64f9406ec1d3" containerID="538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64" exitCode=0 Sep 09 17:27:14.545903 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.545744 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75d68fd4f8-b6krv" Sep 09 17:27:14.545903 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.545748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d68fd4f8-b6krv" event={"ID":"84fcab89-ebed-4708-8b6c-64f9406ec1d3","Type":"ContainerDied","Data":"538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64"} Sep 09 17:27:14.545903 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.545807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75d68fd4f8-b6krv" event={"ID":"84fcab89-ebed-4708-8b6c-64f9406ec1d3","Type":"ContainerDied","Data":"c1db7f68e798148bf97f7ed8897ab16b07a10c10fa163e6e280ce17874a478ad"} Sep 09 17:27:14.545903 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.545835 2580 scope.go:117] "RemoveContainer" containerID="538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64" Sep 09 17:27:14.554702 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.554683 2580 scope.go:117] "RemoveContainer" containerID="538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64" Sep 09 17:27:14.555018 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:27:14.554997 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64\": container with ID starting with 538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64 not found: ID does not exist" containerID="538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64" Sep 09 17:27:14.555102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.555023 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64"} err="failed to get container status \"538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64\": rpc error: code = NotFound desc = could not find container \"538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64\": container with ID starting with 538fe9721f845ebffcf2c434e79c70c5bfde14d6eb36a572b4796ce0b6cd1d64 not found: ID does not exist" Sep 09 17:27:14.568225 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.568203 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75d68fd4f8-b6krv"] Sep 09 17:27:14.572167 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:14.572148 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75d68fd4f8-b6krv"] Sep 09 17:27:16.134632 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:16.134599 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84fcab89-ebed-4708-8b6c-64f9406ec1d3" path="/var/lib/kubelet/pods/84fcab89-ebed-4708-8b6c-64f9406ec1d3/volumes" Sep 09 17:27:17.227134 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.227098 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d7468889-l5s9n"] Sep 09 17:27:17.227570 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.227543 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84fcab89-ebed-4708-8b6c-64f9406ec1d3" containerName="console" Sep 09 17:27:17.227570 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.227557 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="84fcab89-ebed-4708-8b6c-64f9406ec1d3" containerName="console" Sep 09 17:27:17.227644 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.227631 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="84fcab89-ebed-4708-8b6c-64f9406ec1d3" containerName="console" Sep 09 17:27:17.232427 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.232404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.235624 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.235599 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 09 17:27:17.235789 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.235666 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 09 17:27:17.236078 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.235923 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 17:27:17.236176 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.236080 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wcppt\"" Sep 09 17:27:17.236741 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.236702 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 17:27:17.241911 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.241868 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d7468889-l5s9n"] Sep 09 17:27:17.249234 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.249211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcrg\" (UniqueName: \"kubernetes.io/projected/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-kube-api-access-cjcrg\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.249363 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.249256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tmp\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.249363 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.249282 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tls-certs\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.350512 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.350485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tmp\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.350694 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.350522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tls-certs\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.350694 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.350575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcrg\" (UniqueName: \"kubernetes.io/projected/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-kube-api-access-cjcrg\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.350694 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:27:17.350673 2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 09 17:27:17.350844 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:27:17.350739 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tls-certs podName:b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a nodeName:}" failed. No retries permitted until 2026-09-09 17:27:17.850724092 +0000 UTC m=+632.199085543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tls-certs") pod "kube-auth-proxy-5d7468889-l5s9n" (UID: "b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a") : secret "kube-auth-proxy-tls" not found Sep 09 17:27:17.352625 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.352599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tmp\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.361460 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.361440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcrg\" (UniqueName: \"kubernetes.io/projected/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-kube-api-access-cjcrg\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.853977 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.853945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tls-certs\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:17.856541 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:17.856515 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a-tls-certs\") pod \"kube-auth-proxy-5d7468889-l5s9n\" (UID: \"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a\") " pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:18.144620 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.144539 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" Sep 09 17:27:18.429006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.428983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d7468889-l5s9n"] Sep 09 17:27:18.431471 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:27:18.431437 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c9b8dc_c002_4aeb_9c06_7f7ae199df7a.slice/crio-50d683e2a04bd8dc25baf5c2441e23913550e7c5abca7585e95f968eb7661b6f WatchSource:0}: Error finding container 50d683e2a04bd8dc25baf5c2441e23913550e7c5abca7585e95f968eb7661b6f: Status 404 returned error can't find the container with id 50d683e2a04bd8dc25baf5c2441e23913550e7c5abca7585e95f968eb7661b6f Sep 09 17:27:18.489668 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.489635 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr"] Sep 09 17:27:18.494064 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.494049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.496771 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.496747 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 09 17:27:18.497110 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.497094 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wdxr9\"" Sep 09 17:27:18.497415 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.497402 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 09 17:27:18.497611 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.497598 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 17:27:18.497954 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.497940 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 09 17:27:18.505030 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.505010 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr"] Sep 09 17:27:18.559230 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.559203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.559343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.559258 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f75060-6905-4d30-a27a-59c7aa79410e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.559343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.559303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.559343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.559334 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.559448 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.559377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.559448 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.559410 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.559448 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.559438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7qbm\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-kube-api-access-g7qbm\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.566140 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.566118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" event={"ID":"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a","Type":"ContainerStarted","Data":"50d683e2a04bd8dc25baf5c2441e23913550e7c5abca7585e95f968eb7661b6f"} Sep 09 17:27:18.660511 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.660488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f75060-6905-4d30-a27a-59c7aa79410e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.660628 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.660524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.660628 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.660540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.660628 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.660559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.660628 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.660582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.660628 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.660613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qbm\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-kube-api-access-g7qbm\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.660887 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.660651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.661239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.661214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.662817 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.662797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f75060-6905-4d30-a27a-59c7aa79410e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.662964 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.662944 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.663035 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.663011 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.663035 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.663024 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.677532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.677513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.677901 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.677882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7qbm\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-kube-api-access-g7qbm\") pod \"istiod-openshift-gateway-7b7995d5cd-zmkhr\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:18.804519 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:18.804492 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:19.077723 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:19.077688 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr"] Sep 09 17:27:19.079618 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:27:19.079590 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f75060_6905_4d30_a27a_59c7aa79410e.slice/crio-902071f04b8aa4e284d4d8950dc0c59631768e44aede0b15f95d55dd5df2b2c3 WatchSource:0}: Error finding container 902071f04b8aa4e284d4d8950dc0c59631768e44aede0b15f95d55dd5df2b2c3: Status 404 returned error can't find the container with id 902071f04b8aa4e284d4d8950dc0c59631768e44aede0b15f95d55dd5df2b2c3 Sep 09 17:27:19.572614 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:19.572576 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" event={"ID":"79f75060-6905-4d30-a27a-59c7aa79410e","Type":"ContainerStarted","Data":"902071f04b8aa4e284d4d8950dc0c59631768e44aede0b15f95d55dd5df2b2c3"} Sep 09 17:27:22.768587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:22.768516 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:27:22.768848 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:22.768586 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:27:23.597065 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:23.597019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" event={"ID":"b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a","Type":"ContainerStarted","Data":"12c2a1fb72ae5fe0faae0724dfb2e747c8b999c66681be75a5817dfb4fb9f283"} Sep 09 17:27:23.598952 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:23.598926 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" event={"ID":"79f75060-6905-4d30-a27a-59c7aa79410e","Type":"ContainerStarted","Data":"2ba41240f3d3bdc949975a0239838e6f5065aeaa97128ba25071297d898c07cb"} Sep 09 17:27:23.599127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:23.599110 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:23.600545 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:23.600520 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-zmkhr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 17:27:23.600676 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:23.600569 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" podUID="79f75060-6905-4d30-a27a-59c7aa79410e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 17:27:23.614891 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:23.614837 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d7468889-l5s9n" podStartSLOduration=2.279417717 podStartE2EDuration="6.614819438s" totalImagesPullingTime="4.335401721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:27:23.61441565 +0000 UTC m=+637.962777124" watchObservedRunningTime="2026-09-09 17:27:23.614819438 +0000 UTC m=+637.963180913" Sep 09 17:27:23.635270 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:23.635226 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" podStartSLOduration=1.94853218 podStartE2EDuration="5.635209114s" totalImagesPullingTime="3.686676934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:27:23.633217769 +0000 UTC m=+637.981579276" watchObservedRunningTime="2026-09-09 17:27:23.635209114 +0000 UTC m=+637.983570588" Sep 09 17:27:24.605343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:24.605315 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:37.228903 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.228870 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5slf"] Sep 09 17:27:37.233575 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.233559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:37.236927 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.236907 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tr89p\"" Sep 09 17:27:37.237089 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.237070 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 09 17:27:37.248784 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.248761 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l"] Sep 09 17:27:37.255015 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.254992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:37.256523 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.256355 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5slf"] Sep 09 17:27:37.266306 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.266271 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 09 17:27:37.266629 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.266607 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9wzl4\"" Sep 09 17:27:37.286889 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.282875 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l"] Sep 09 17:27:37.324377 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.324339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-szp6l\" (UID: \"19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:37.324609 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.324409 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529ad62f-ef17-4186-8b23-45a601d9bfef-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5slf\" (UID: \"529ad62f-ef17-4186-8b23-45a601d9bfef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:37.324609 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.324457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckznf\" (UniqueName: \"kubernetes.io/projected/529ad62f-ef17-4186-8b23-45a601d9bfef-kube-api-access-ckznf\") pod \"kserve-controller-manager-d6f4b65b8-n5slf\" (UID: \"529ad62f-ef17-4186-8b23-45a601d9bfef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:37.324609 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.324484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vh4d\" (UniqueName: \"kubernetes.io/projected/19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b-kube-api-access-4vh4d\") pod \"llmisvc-controller-manager-777c6bd7d5-szp6l\" (UID: \"19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:37.425367 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.425332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-szp6l\" (UID: \"19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:37.425588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.425395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529ad62f-ef17-4186-8b23-45a601d9bfef-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5slf\" (UID: \"529ad62f-ef17-4186-8b23-45a601d9bfef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:37.425588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.425423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckznf\" (UniqueName: \"kubernetes.io/projected/529ad62f-ef17-4186-8b23-45a601d9bfef-kube-api-access-ckznf\") pod \"kserve-controller-manager-d6f4b65b8-n5slf\" (UID: \"529ad62f-ef17-4186-8b23-45a601d9bfef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:37.425588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.425452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vh4d\" (UniqueName: \"kubernetes.io/projected/19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b-kube-api-access-4vh4d\") pod \"llmisvc-controller-manager-777c6bd7d5-szp6l\" (UID: \"19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:37.425588 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:27:37.425534 2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 09 17:27:37.425803 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:27:37.425596 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/529ad62f-ef17-4186-8b23-45a601d9bfef-cert podName:529ad62f-ef17-4186-8b23-45a601d9bfef nodeName:}" failed. No retries permitted until 2026-09-09 17:27:37.925580999 +0000 UTC m=+652.273942453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/529ad62f-ef17-4186-8b23-45a601d9bfef-cert") pod "kserve-controller-manager-d6f4b65b8-n5slf" (UID: "529ad62f-ef17-4186-8b23-45a601d9bfef") : secret "kserve-webhook-server-cert" not found Sep 09 17:27:37.427820 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.427789 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-szp6l\" (UID: \"19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:37.439940 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.439914 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckznf\" (UniqueName: \"kubernetes.io/projected/529ad62f-ef17-4186-8b23-45a601d9bfef-kube-api-access-ckznf\") pod \"kserve-controller-manager-d6f4b65b8-n5slf\" (UID: \"529ad62f-ef17-4186-8b23-45a601d9bfef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:37.440071 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.439962 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vh4d\" (UniqueName: \"kubernetes.io/projected/19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b-kube-api-access-4vh4d\") pod \"llmisvc-controller-manager-777c6bd7d5-szp6l\" (UID: \"19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:37.593791 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.593741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:37.866567 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.866537 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l"] Sep 09 17:27:37.868953 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:27:37.868927 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod19cebdd2_d398_40ba_bc4d_ab2f7a98ce5b.slice/crio-4fcac005e61e6a89e3aae1b83e0748edd56da9dbfc2236ec4eb4a1c40beb8367 WatchSource:0}: Error finding container 4fcac005e61e6a89e3aae1b83e0748edd56da9dbfc2236ec4eb4a1c40beb8367: Status 404 returned error can't find the container with id 4fcac005e61e6a89e3aae1b83e0748edd56da9dbfc2236ec4eb4a1c40beb8367 Sep 09 17:27:37.929448 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.929405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529ad62f-ef17-4186-8b23-45a601d9bfef-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5slf\" (UID: \"529ad62f-ef17-4186-8b23-45a601d9bfef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:37.931793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:37.931772 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/529ad62f-ef17-4186-8b23-45a601d9bfef-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5slf\" (UID: \"529ad62f-ef17-4186-8b23-45a601d9bfef\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:38.143427 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.143330 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:38.415629 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.415603 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5slf"] Sep 09 17:27:38.417653 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:27:38.417613 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529ad62f_ef17_4186_8b23_45a601d9bfef.slice/crio-a3326502fcb50837b847d37dfaaf4483ac3e44f135d372f9b046c05f9354b3f8 WatchSource:0}: Error finding container a3326502fcb50837b847d37dfaaf4483ac3e44f135d372f9b046c05f9354b3f8: Status 404 returned error can't find the container with id a3326502fcb50837b847d37dfaaf4483ac3e44f135d372f9b046c05f9354b3f8 Sep 09 17:27:38.439557 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.439529 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr"] Sep 09 17:27:38.439803 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.439774 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" podUID="79f75060-6905-4d30-a27a-59c7aa79410e" containerName="discovery" containerID="cri-o://2ba41240f3d3bdc949975a0239838e6f5065aeaa97128ba25071297d898c07cb" gracePeriod=30 Sep 09 17:27:38.680126 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.680092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" event={"ID":"19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b","Type":"ContainerStarted","Data":"4fcac005e61e6a89e3aae1b83e0748edd56da9dbfc2236ec4eb4a1c40beb8367"} Sep 09 17:27:38.681669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.681644 2580 generic.go:356] "Generic (PLEG): container finished" podID="79f75060-6905-4d30-a27a-59c7aa79410e" containerID="2ba41240f3d3bdc949975a0239838e6f5065aeaa97128ba25071297d898c07cb" exitCode=0 Sep 09 17:27:38.681879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.681801 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" event={"ID":"79f75060-6905-4d30-a27a-59c7aa79410e","Type":"ContainerDied","Data":"2ba41240f3d3bdc949975a0239838e6f5065aeaa97128ba25071297d898c07cb"} Sep 09 17:27:38.681879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.681857 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" event={"ID":"79f75060-6905-4d30-a27a-59c7aa79410e","Type":"ContainerDied","Data":"902071f04b8aa4e284d4d8950dc0c59631768e44aede0b15f95d55dd5df2b2c3"} Sep 09 17:27:38.681982 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.681879 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="902071f04b8aa4e284d4d8950dc0c59631768e44aede0b15f95d55dd5df2b2c3" Sep 09 17:27:38.682967 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.682946 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" event={"ID":"529ad62f-ef17-4186-8b23-45a601d9bfef","Type":"ContainerStarted","Data":"a3326502fcb50837b847d37dfaaf4483ac3e44f135d372f9b046c05f9354b3f8"} Sep 09 17:27:38.687794 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.687774 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:38.737243 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737213 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-dns-cert\") pod \"79f75060-6905-4d30-a27a-59c7aa79410e\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " Sep 09 17:27:38.737403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737250 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-kubeconfig\") pod \"79f75060-6905-4d30-a27a-59c7aa79410e\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " Sep 09 17:27:38.737403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737307 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-istio-token\") pod \"79f75060-6905-4d30-a27a-59c7aa79410e\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " Sep 09 17:27:38.737403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737335 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-ca-configmap\") pod \"79f75060-6905-4d30-a27a-59c7aa79410e\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " Sep 09 17:27:38.737403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737360 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7qbm\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-kube-api-access-g7qbm\") pod \"79f75060-6905-4d30-a27a-59c7aa79410e\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " Sep 09 17:27:38.737666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737437 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-cacerts\") pod \"79f75060-6905-4d30-a27a-59c7aa79410e\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " Sep 09 17:27:38.737666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737542 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f75060-6905-4d30-a27a-59c7aa79410e-local-certs\") pod \"79f75060-6905-4d30-a27a-59c7aa79410e\" (UID: \"79f75060-6905-4d30-a27a-59c7aa79410e\") " Sep 09 17:27:38.737804 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737751 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "79f75060-6905-4d30-a27a-59c7aa79410e" (UID: "79f75060-6905-4d30-a27a-59c7aa79410e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:27:38.737949 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.737907 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-ca-configmap\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:38.739899 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.739843 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-kube-api-access-g7qbm" (OuterVolumeSpecName: "kube-api-access-g7qbm") pod "79f75060-6905-4d30-a27a-59c7aa79410e" (UID: "79f75060-6905-4d30-a27a-59c7aa79410e"). InnerVolumeSpecName "kube-api-access-g7qbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:27:38.740007 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.739929 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-cacerts" (OuterVolumeSpecName: "cacerts") pod "79f75060-6905-4d30-a27a-59c7aa79410e" (UID: "79f75060-6905-4d30-a27a-59c7aa79410e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:27:38.740077 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.740006 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f75060-6905-4d30-a27a-59c7aa79410e-local-certs" (OuterVolumeSpecName: "local-certs") pod "79f75060-6905-4d30-a27a-59c7aa79410e" (UID: "79f75060-6905-4d30-a27a-59c7aa79410e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:27:38.740077 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.740021 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "79f75060-6905-4d30-a27a-59c7aa79410e" (UID: "79f75060-6905-4d30-a27a-59c7aa79410e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:27:38.740175 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.740127 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-istio-token" (OuterVolumeSpecName: "istio-token") pod "79f75060-6905-4d30-a27a-59c7aa79410e" (UID: "79f75060-6905-4d30-a27a-59c7aa79410e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:27:38.740175 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.740150 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "79f75060-6905-4d30-a27a-59c7aa79410e" (UID: "79f75060-6905-4d30-a27a-59c7aa79410e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:27:38.838809 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.838775 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-cacerts\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:38.838809 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.838804 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f75060-6905-4d30-a27a-59c7aa79410e-local-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:38.838809 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.838815 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-csr-dns-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:38.839039 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.838824 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f75060-6905-4d30-a27a-59c7aa79410e-istio-kubeconfig\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:38.839039 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.838834 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:38.839039 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:38.838843 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7qbm\" (UniqueName: \"kubernetes.io/projected/79f75060-6905-4d30-a27a-59c7aa79410e-kube-api-access-g7qbm\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:27:39.688876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:39.688844 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr" Sep 09 17:27:39.712763 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:39.712725 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr"] Sep 09 17:27:39.720046 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:39.720005 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-zmkhr"] Sep 09 17:27:40.130409 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:40.130374 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79f75060-6905-4d30-a27a-59c7aa79410e" path="/var/lib/kubelet/pods/79f75060-6905-4d30-a27a-59c7aa79410e/volumes" Sep 09 17:27:41.700466 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:41.700360 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" event={"ID":"529ad62f-ef17-4186-8b23-45a601d9bfef","Type":"ContainerStarted","Data":"7e47500af43511f98cc6b650bccb2e728da99a186ce5ffff4df62d0e55695064"} Sep 09 17:27:41.700878 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:41.700487 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:27:41.701938 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:41.701914 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" event={"ID":"19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b","Type":"ContainerStarted","Data":"31039b912e8ad50b2e7d99197eaf37976a74f982f36ffb651e120b7bd46d968e"} Sep 09 17:27:41.702058 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:41.701961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:27:41.719664 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:41.719612 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" podStartSLOduration=1.870167036 podStartE2EDuration="4.719597493s" totalImagesPullingTime="2.849430457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:27:41.718164561 +0000 UTC m=+656.066526033" watchObservedRunningTime="2026-09-09 17:27:41.719597493 +0000 UTC m=+656.067958966" Sep 09 17:27:41.736826 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:27:41.736775 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" podStartSLOduration=1.341924367 podStartE2EDuration="4.736759113s" totalImagesPullingTime="3.394834746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:27:41.735074727 +0000 UTC m=+656.083436199" watchObservedRunningTime="2026-09-09 17:27:41.736759113 +0000 UTC m=+656.085120586" Sep 09 17:28:12.708540 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:12.708505 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-szp6l" Sep 09 17:28:12.711666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:12.711645 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5slf" Sep 09 17:28:44.801985 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.801904 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx"] Sep 09 17:28:44.802419 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.802261 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79f75060-6905-4d30-a27a-59c7aa79410e" containerName="discovery" Sep 09 17:28:44.802419 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.802272 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79f75060-6905-4d30-a27a-59c7aa79410e" containerName="discovery" Sep 09 17:28:44.802419 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.802342 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79f75060-6905-4d30-a27a-59c7aa79410e" containerName="discovery" Sep 09 17:28:44.805332 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.805318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:28:44.807914 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.807889 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 17:28:44.809143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.809124 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 17:28:44.809236 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.809124 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mdbrq\"" Sep 09 17:28:44.814434 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.814410 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx"] Sep 09 17:28:44.899542 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:44.899514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6cd\" (UniqueName: \"kubernetes.io/projected/e7d6641d-2b36-45e1-a371-625a1b730e4b-kube-api-access-kz6cd\") pod \"limitador-operator-controller-manager-74495c69f8-5dgxx\" (UID: \"e7d6641d-2b36-45e1-a371-625a1b730e4b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:28:45.001104 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:45.001046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6cd\" (UniqueName: \"kubernetes.io/projected/e7d6641d-2b36-45e1-a371-625a1b730e4b-kube-api-access-kz6cd\") pod \"limitador-operator-controller-manager-74495c69f8-5dgxx\" (UID: \"e7d6641d-2b36-45e1-a371-625a1b730e4b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:28:45.019899 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:45.019878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6cd\" (UniqueName: \"kubernetes.io/projected/e7d6641d-2b36-45e1-a371-625a1b730e4b-kube-api-access-kz6cd\") pod \"limitador-operator-controller-manager-74495c69f8-5dgxx\" (UID: \"e7d6641d-2b36-45e1-a371-625a1b730e4b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:28:45.115967 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:45.115882 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:28:45.398091 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:45.398067 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx"] Sep 09 17:28:45.403753 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:28:45.400998 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d6641d_2b36_45e1_a371_625a1b730e4b.slice/crio-8be6e5efb3319e6816c4eb243be312f1a3de0b3d4e9e605510fa397d875bc852 WatchSource:0}: Error finding container 8be6e5efb3319e6816c4eb243be312f1a3de0b3d4e9e605510fa397d875bc852: Status 404 returned error can't find the container with id 8be6e5efb3319e6816c4eb243be312f1a3de0b3d4e9e605510fa397d875bc852 Sep 09 17:28:46.010397 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:46.010363 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" event={"ID":"e7d6641d-2b36-45e1-a371-625a1b730e4b","Type":"ContainerStarted","Data":"8be6e5efb3319e6816c4eb243be312f1a3de0b3d4e9e605510fa397d875bc852"} Sep 09 17:28:48.927961 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:48.927925 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr"] Sep 09 17:28:48.931736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:48.931679 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:48.934296 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:48.934275 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9rfkr\"" Sep 09 17:28:48.944334 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:48.944313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr"] Sep 09 17:28:49.027858 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.027824 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" event={"ID":"e7d6641d-2b36-45e1-a371-625a1b730e4b","Type":"ContainerStarted","Data":"19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e"} Sep 09 17:28:49.028005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.027902 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:28:49.032967 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.032947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" (UID: \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:49.033064 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.032980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556r9\" (UniqueName: \"kubernetes.io/projected/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-kube-api-access-556r9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" (UID: \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:49.049662 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.049625 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" podStartSLOduration=2.4770154 podStartE2EDuration="5.049612969s" totalImagesPullingTime="2.572597569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:28:49.049357219 +0000 UTC m=+723.397718694" watchObservedRunningTime="2026-09-09 17:28:49.049612969 +0000 UTC m=+723.397974444" Sep 09 17:28:49.133599 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.133573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" (UID: \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:49.133706 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.133616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556r9\" (UniqueName: \"kubernetes.io/projected/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-kube-api-access-556r9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" (UID: \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:49.133938 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.133920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" (UID: \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:49.141534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.141513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-556r9\" (UniqueName: \"kubernetes.io/projected/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-kube-api-access-556r9\") pod \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" (UID: \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:49.241224 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.241174 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:49.516281 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:49.516254 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr"] Sep 09 17:28:49.518286 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:28:49.518259 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f8d03f_3f24_4a86_b8ca_d7a1063f5916.slice/crio-5fb78d6ddbfbde55bd116210478f1e38f57cce99dfe769a7c0c22363e37a13ab WatchSource:0}: Error finding container 5fb78d6ddbfbde55bd116210478f1e38f57cce99dfe769a7c0c22363e37a13ab: Status 404 returned error can't find the container with id 5fb78d6ddbfbde55bd116210478f1e38f57cce99dfe769a7c0c22363e37a13ab Sep 09 17:28:50.036075 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:50.036040 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" event={"ID":"12f8d03f-3f24-4a86-b8ca-d7a1063f5916","Type":"ContainerStarted","Data":"5fb78d6ddbfbde55bd116210478f1e38f57cce99dfe769a7c0c22363e37a13ab"} Sep 09 17:28:51.650249 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:51.650213 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qcjwm"] Sep 09 17:28:51.654211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:51.654188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" Sep 09 17:28:51.656949 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:51.656926 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4lzpc\"" Sep 09 17:28:51.665070 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:51.665045 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qcjwm"] Sep 09 17:28:51.755170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:51.755134 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6npl\" (UniqueName: \"kubernetes.io/projected/49626bf1-e673-452a-8f0f-5f84adf7a9d3-kube-api-access-w6npl\") pod \"authorino-operator-76b478bd78-qcjwm\" (UID: \"49626bf1-e673-452a-8f0f-5f84adf7a9d3\") " pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" Sep 09 17:28:51.856289 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:51.856252 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6npl\" (UniqueName: \"kubernetes.io/projected/49626bf1-e673-452a-8f0f-5f84adf7a9d3-kube-api-access-w6npl\") pod \"authorino-operator-76b478bd78-qcjwm\" (UID: \"49626bf1-e673-452a-8f0f-5f84adf7a9d3\") " pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" Sep 09 17:28:51.868167 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:51.868137 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6npl\" (UniqueName: \"kubernetes.io/projected/49626bf1-e673-452a-8f0f-5f84adf7a9d3-kube-api-access-w6npl\") pod \"authorino-operator-76b478bd78-qcjwm\" (UID: \"49626bf1-e673-452a-8f0f-5f84adf7a9d3\") " pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" Sep 09 17:28:51.968992 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:51.968912 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" Sep 09 17:28:52.256806 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:52.256773 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-qcjwm"] Sep 09 17:28:52.259219 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:28:52.259170 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49626bf1_e673_452a_8f0f_5f84adf7a9d3.slice/crio-956741dcf6f37bd93d5d1fadaf2c39fe94f4e5acb3fb0ed6dcb4eb8a2ead2bad WatchSource:0}: Error finding container 956741dcf6f37bd93d5d1fadaf2c39fe94f4e5acb3fb0ed6dcb4eb8a2ead2bad: Status 404 returned error can't find the container with id 956741dcf6f37bd93d5d1fadaf2c39fe94f4e5acb3fb0ed6dcb4eb8a2ead2bad Sep 09 17:28:53.057681 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:53.057645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" event={"ID":"49626bf1-e673-452a-8f0f-5f84adf7a9d3","Type":"ContainerStarted","Data":"956741dcf6f37bd93d5d1fadaf2c39fe94f4e5acb3fb0ed6dcb4eb8a2ead2bad"} Sep 09 17:28:53.961916 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:53.961821 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q"] Sep 09 17:28:53.967934 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:53.967432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" Sep 09 17:28:53.971403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:53.971041 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-hd2gc\"" Sep 09 17:28:53.971403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:53.971051 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 09 17:28:53.986540 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:53.986506 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q"] Sep 09 17:28:54.066999 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:54.066958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" event={"ID":"12f8d03f-3f24-4a86-b8ca-d7a1063f5916","Type":"ContainerStarted","Data":"c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b"} Sep 09 17:28:54.067505 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:54.067066 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:28:54.080112 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:54.080080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dr4w\" (UniqueName: \"kubernetes.io/projected/db0b660e-bbbf-44cb-9ddc-aa196c8ceddd-kube-api-access-8dr4w\") pod \"dns-operator-controller-manager-7b84cd7c9c-vpb4q\" (UID: \"db0b660e-bbbf-44cb-9ddc-aa196c8ceddd\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" Sep 09 17:28:54.181189 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:54.181146 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dr4w\" (UniqueName: \"kubernetes.io/projected/db0b660e-bbbf-44cb-9ddc-aa196c8ceddd-kube-api-access-8dr4w\") pod \"dns-operator-controller-manager-7b84cd7c9c-vpb4q\" (UID: \"db0b660e-bbbf-44cb-9ddc-aa196c8ceddd\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" Sep 09 17:28:54.189208 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:54.189179 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dr4w\" (UniqueName: \"kubernetes.io/projected/db0b660e-bbbf-44cb-9ddc-aa196c8ceddd-kube-api-access-8dr4w\") pod \"dns-operator-controller-manager-7b84cd7c9c-vpb4q\" (UID: \"db0b660e-bbbf-44cb-9ddc-aa196c8ceddd\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" Sep 09 17:28:54.284031 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:54.283994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" Sep 09 17:28:54.553678 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:54.553632 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" podStartSLOduration=2.5160585109999998 podStartE2EDuration="6.553613164s" totalImagesPullingTime="4.037554653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:28:54.08774807 +0000 UTC m=+728.436109544" watchObservedRunningTime="2026-09-09 17:28:54.553613164 +0000 UTC m=+728.901974631" Sep 09 17:28:54.554135 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:54.554117 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q"] Sep 09 17:28:54.556554 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:28:54.556528 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0b660e_bbbf_44cb_9ddc_aa196c8ceddd.slice/crio-0e3b06cec34c9ab946406ff61a60d82bff67eda89dc12009ece5af1fcd02c5ac WatchSource:0}: Error finding container 0e3b06cec34c9ab946406ff61a60d82bff67eda89dc12009ece5af1fcd02c5ac: Status 404 returned error can't find the container with id 0e3b06cec34c9ab946406ff61a60d82bff67eda89dc12009ece5af1fcd02c5ac Sep 09 17:28:55.074438 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:55.074396 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" event={"ID":"db0b660e-bbbf-44cb-9ddc-aa196c8ceddd","Type":"ContainerStarted","Data":"0e3b06cec34c9ab946406ff61a60d82bff67eda89dc12009ece5af1fcd02c5ac"} Sep 09 17:28:56.170541 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.170510 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2"] Sep 09 17:28:56.174846 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.174819 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.177333 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.177310 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 09 17:28:56.177442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.177365 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 09 17:28:56.177442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.177376 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-q6dcm\"" Sep 09 17:28:56.181223 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.181203 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2"] Sep 09 17:28:56.299180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.299144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5sr\" (UniqueName: \"kubernetes.io/projected/6c4eee49-a401-44f5-9d03-9a87269ba67c-kube-api-access-5d5sr\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.299361 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.299217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c4eee49-a401-44f5-9d03-9a87269ba67c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.299361 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.299320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c4eee49-a401-44f5-9d03-9a87269ba67c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.400522 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.400486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c4eee49-a401-44f5-9d03-9a87269ba67c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.400682 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.400582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c4eee49-a401-44f5-9d03-9a87269ba67c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.400682 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:28:56.400637 2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 09 17:28:56.400682 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.400658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5sr\" (UniqueName: \"kubernetes.io/projected/6c4eee49-a401-44f5-9d03-9a87269ba67c-kube-api-access-5d5sr\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.400885 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:28:56.400730 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c4eee49-a401-44f5-9d03-9a87269ba67c-plugin-serving-cert podName:6c4eee49-a401-44f5-9d03-9a87269ba67c nodeName:}" failed. No retries permitted until 2026-09-09 17:28:56.900688529 +0000 UTC m=+731.249050001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/6c4eee49-a401-44f5-9d03-9a87269ba67c-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-m5rs2" (UID: "6c4eee49-a401-44f5-9d03-9a87269ba67c") : secret "plugin-serving-cert" not found Sep 09 17:28:56.401400 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.401371 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6c4eee49-a401-44f5-9d03-9a87269ba67c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.409772 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.409748 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5sr\" (UniqueName: \"kubernetes.io/projected/6c4eee49-a401-44f5-9d03-9a87269ba67c-kube-api-access-5d5sr\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.904882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.904853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c4eee49-a401-44f5-9d03-9a87269ba67c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:56.907207 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:56.907188 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6c4eee49-a401-44f5-9d03-9a87269ba67c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m5rs2\" (UID: \"6c4eee49-a401-44f5-9d03-9a87269ba67c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:57.086427 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:57.086398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" Sep 09 17:28:57.088282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:57.088256 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" event={"ID":"db0b660e-bbbf-44cb-9ddc-aa196c8ceddd","Type":"ContainerStarted","Data":"590cce3ec46888085de4a186cbd0ac58e618c7c8b566c4dcbcda87d20b22a095"} Sep 09 17:28:57.088382 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:57.088326 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" Sep 09 17:28:57.090353 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:57.090321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" event={"ID":"49626bf1-e673-452a-8f0f-5f84adf7a9d3","Type":"ContainerStarted","Data":"b7e07c0ebc462ea660f5ac2c8e9d8631f12ed388e5ae7373d039e0664595ef4b"} Sep 09 17:28:57.090594 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:57.090425 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" Sep 09 17:28:57.109685 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:57.109644 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" podStartSLOduration=2.174469934 podStartE2EDuration="4.109630159s" totalImagesPullingTime="1.935160225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:28:57.10724624 +0000 UTC m=+731.455607714" watchObservedRunningTime="2026-09-09 17:28:57.109630159 +0000 UTC m=+731.457991631" Sep 09 17:28:57.124741 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:57.124684 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" podStartSLOduration=1.484674216 podStartE2EDuration="6.124670983s" totalImagesPullingTime="4.639996767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:28:57.122881178 +0000 UTC m=+731.471242652" watchObservedRunningTime="2026-09-09 17:28:57.124670983 +0000 UTC m=+731.473032447" Sep 09 17:28:57.360272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:57.360208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2"] Sep 09 17:28:57.362523 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:28:57.362496 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c4eee49_a401_44f5_9d03_9a87269ba67c.slice/crio-519efa3c65a35bd2e43f94b7243c12ec2bc87c1d120acab78201349162545999 WatchSource:0}: Error finding container 519efa3c65a35bd2e43f94b7243c12ec2bc87c1d120acab78201349162545999: Status 404 returned error can't find the container with id 519efa3c65a35bd2e43f94b7243c12ec2bc87c1d120acab78201349162545999 Sep 09 17:28:58.096759 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:28:58.096722 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" event={"ID":"6c4eee49-a401-44f5-9d03-9a87269ba67c","Type":"ContainerStarted","Data":"519efa3c65a35bd2e43f94b7243c12ec2bc87c1d120acab78201349162545999"} Sep 09 17:29:00.039085 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:00.039051 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:03.124944 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:03.124913 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" event={"ID":"6c4eee49-a401-44f5-9d03-9a87269ba67c","Type":"ContainerStarted","Data":"267dd86d76367da1bad7a531609c66f87fa62526ec7a6545b284d7375ea5481e"} Sep 09 17:29:03.147614 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:03.147570 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m5rs2" podStartSLOduration=2.394559437 podStartE2EDuration="7.147555318s" totalImagesPullingTime="4.752995881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:03.146169708 +0000 UTC m=+737.494531180" watchObservedRunningTime="2026-09-09 17:29:03.147555318 +0000 UTC m=+737.495916791" Sep 09 17:29:05.076355 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:05.076322 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:06.528966 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.528914 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq"] Sep 09 17:29:06.533296 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.533274 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:06.548013 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.547988 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq"] Sep 09 17:29:06.585428 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.585404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgqn\" (UniqueName: \"kubernetes.io/projected/d4514f5c-da70-418b-a681-8ea4585d2392-kube-api-access-dqgqn\") pod \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" (UID: \"d4514f5c-da70-418b-a681-8ea4585d2392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:06.585550 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.585509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4514f5c-da70-418b-a681-8ea4585d2392-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" (UID: \"d4514f5c-da70-418b-a681-8ea4585d2392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:06.686502 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.686476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgqn\" (UniqueName: \"kubernetes.io/projected/d4514f5c-da70-418b-a681-8ea4585d2392-kube-api-access-dqgqn\") pod \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" (UID: \"d4514f5c-da70-418b-a681-8ea4585d2392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:06.686634 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.686546 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4514f5c-da70-418b-a681-8ea4585d2392-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" (UID: \"d4514f5c-da70-418b-a681-8ea4585d2392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:06.686919 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.686901 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4514f5c-da70-418b-a681-8ea4585d2392-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" (UID: \"d4514f5c-da70-418b-a681-8ea4585d2392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:06.713041 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.713015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgqn\" (UniqueName: \"kubernetes.io/projected/d4514f5c-da70-418b-a681-8ea4585d2392-kube-api-access-dqgqn\") pod \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" (UID: \"d4514f5c-da70-418b-a681-8ea4585d2392\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:06.843730 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:06.843686 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:07.124023 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.123994 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq"] Sep 09 17:29:07.125785 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:29:07.125745 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4514f5c_da70_418b_a681_8ea4585d2392.slice/crio-52d0a14a45fbe5d16b31d2d81a3c9d37a1d183676f287319eec0209fd2b6a740 WatchSource:0}: Error finding container 52d0a14a45fbe5d16b31d2d81a3c9d37a1d183676f287319eec0209fd2b6a740: Status 404 returned error can't find the container with id 52d0a14a45fbe5d16b31d2d81a3c9d37a1d183676f287319eec0209fd2b6a740 Sep 09 17:29:07.156255 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.156233 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" event={"ID":"d4514f5c-da70-418b-a681-8ea4585d2392","Type":"ContainerStarted","Data":"52d0a14a45fbe5d16b31d2d81a3c9d37a1d183676f287319eec0209fd2b6a740"} Sep 09 17:29:07.350873 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.350829 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq"] Sep 09 17:29:07.355465 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.355434 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq"] Sep 09 17:29:07.364367 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.364335 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr"] Sep 09 17:29:07.364561 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.364533 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" containerName="manager" containerID="cri-o://c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b" gracePeriod=2 Sep 09 17:29:07.375787 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.375758 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr"] Sep 09 17:29:07.377665 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.377643 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c"] Sep 09 17:29:07.378032 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.378017 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" containerName="manager" Sep 09 17:29:07.378032 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.378031 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" containerName="manager" Sep 09 17:29:07.378161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.378047 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4514f5c-da70-418b-a681-8ea4585d2392" containerName="manager" Sep 09 17:29:07.378161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.378053 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4514f5c-da70-418b-a681-8ea4585d2392" containerName="manager" Sep 09 17:29:07.378161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.378117 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" containerName="manager" Sep 09 17:29:07.378161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.378126 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4514f5c-da70-418b-a681-8ea4585d2392" containerName="manager" Sep 09 17:29:07.381060 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.381041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:07.384462 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.384424 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-568hr\": pods \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:07.391210 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.391185 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c"] Sep 09 17:29:07.391495 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.391475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3edc23f1-c7c2-448e-a6e6-65717671535b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-psz2c\" (UID: \"3edc23f1-c7c2-448e-a6e6-65717671535b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:07.391573 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.391559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q29m\" (UniqueName: \"kubernetes.io/projected/3edc23f1-c7c2-448e-a6e6-65717671535b-kube-api-access-7q29m\") pod \"kuadrant-operator-controller-manager-5f685c496f-psz2c\" (UID: \"3edc23f1-c7c2-448e-a6e6-65717671535b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:07.395660 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.395640 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx"] Sep 09 17:29:07.395840 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.395822 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" containerName="manager" containerID="cri-o://19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e" gracePeriod=2 Sep 09 17:29:07.413632 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.410516 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz"] Sep 09 17:29:07.417604 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.417493 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:07.417821 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.417793 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx"] Sep 09 17:29:07.417989 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.417966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:07.420093 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.420073 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz"] Sep 09 17:29:07.420198 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.420067 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5dgxx\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:07.421724 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.421692 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96"] Sep 09 17:29:07.422092 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.422078 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" containerName="manager" Sep 09 17:29:07.422158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.422094 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" containerName="manager" Sep 09 17:29:07.422207 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.422161 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" containerName="manager" Sep 09 17:29:07.422288 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.422268 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:07.425956 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.425937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" Sep 09 17:29:07.432226 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.432207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96"] Sep 09 17:29:07.447980 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.447953 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5dgxx\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:07.449830 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.449807 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:07.469683 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.469653 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5dgxx\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:07.472127 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.472092 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:07.492014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.491989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3edc23f1-c7c2-448e-a6e6-65717671535b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-psz2c\" (UID: \"3edc23f1-c7c2-448e-a6e6-65717671535b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:07.492136 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.492043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4sx\" (UniqueName: \"kubernetes.io/projected/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-kube-api-access-zn4sx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42cxz\" (UID: \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:07.492136 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.492061 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdvw6\" (UniqueName: \"kubernetes.io/projected/9b29873e-daf4-4b8c-93a6-77a0b0464522-kube-api-access-mdvw6\") pod \"limitador-operator-controller-manager-74495c69f8-swb96\" (UID: \"9b29873e-daf4-4b8c-93a6-77a0b0464522\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" Sep 09 17:29:07.492136 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.492087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42cxz\" (UID: \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:07.492136 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.492111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q29m\" (UniqueName: \"kubernetes.io/projected/3edc23f1-c7c2-448e-a6e6-65717671535b-kube-api-access-7q29m\") pod \"kuadrant-operator-controller-manager-5f685c496f-psz2c\" (UID: \"3edc23f1-c7c2-448e-a6e6-65717671535b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:07.492301 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.492284 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3edc23f1-c7c2-448e-a6e6-65717671535b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-psz2c\" (UID: \"3edc23f1-c7c2-448e-a6e6-65717671535b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:07.500729 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.500698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q29m\" (UniqueName: \"kubernetes.io/projected/3edc23f1-c7c2-448e-a6e6-65717671535b-kube-api-access-7q29m\") pod \"kuadrant-operator-controller-manager-5f685c496f-psz2c\" (UID: \"3edc23f1-c7c2-448e-a6e6-65717671535b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:07.593414 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.593376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4sx\" (UniqueName: \"kubernetes.io/projected/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-kube-api-access-zn4sx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42cxz\" (UID: \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:07.593882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.593425 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdvw6\" (UniqueName: \"kubernetes.io/projected/9b29873e-daf4-4b8c-93a6-77a0b0464522-kube-api-access-mdvw6\") pod \"limitador-operator-controller-manager-74495c69f8-swb96\" (UID: \"9b29873e-daf4-4b8c-93a6-77a0b0464522\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" Sep 09 17:29:07.593882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.593464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42cxz\" (UID: \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:07.593998 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.593876 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42cxz\" (UID: \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:07.604925 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.604895 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4sx\" (UniqueName: \"kubernetes.io/projected/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-kube-api-access-zn4sx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-42cxz\" (UID: \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:07.605059 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.605001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdvw6\" (UniqueName: \"kubernetes.io/projected/9b29873e-daf4-4b8c-93a6-77a0b0464522-kube-api-access-mdvw6\") pod \"limitador-operator-controller-manager-74495c69f8-swb96\" (UID: \"9b29873e-daf4-4b8c-93a6-77a0b0464522\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" Sep 09 17:29:07.618048 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.618030 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:07.620585 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.620555 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5dgxx\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:07.622748 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.622705 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:07.624979 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.624936 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:07.627158 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.627136 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5dgxx\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:07.629082 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:07.629061 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:07.694473 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.694450 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-extensions-socket-volume\") pod \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\" (UID: \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\") " Sep 09 17:29:07.694569 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.694521 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz6cd\" (UniqueName: \"kubernetes.io/projected/e7d6641d-2b36-45e1-a371-625a1b730e4b-kube-api-access-kz6cd\") pod \"e7d6641d-2b36-45e1-a371-625a1b730e4b\" (UID: \"e7d6641d-2b36-45e1-a371-625a1b730e4b\") " Sep 09 17:29:07.694569 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.694566 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-556r9\" (UniqueName: \"kubernetes.io/projected/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-kube-api-access-556r9\") pod \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\" (UID: \"12f8d03f-3f24-4a86-b8ca-d7a1063f5916\") " Sep 09 17:29:07.694924 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.694904 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "12f8d03f-3f24-4a86-b8ca-d7a1063f5916" (UID: "12f8d03f-3f24-4a86-b8ca-d7a1063f5916"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:29:07.696597 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.696576 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d6641d-2b36-45e1-a371-625a1b730e4b-kube-api-access-kz6cd" (OuterVolumeSpecName: "kube-api-access-kz6cd") pod "e7d6641d-2b36-45e1-a371-625a1b730e4b" (UID: "e7d6641d-2b36-45e1-a371-625a1b730e4b"). InnerVolumeSpecName "kube-api-access-kz6cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:29:07.696654 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.696599 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-kube-api-access-556r9" (OuterVolumeSpecName: "kube-api-access-556r9") pod "12f8d03f-3f24-4a86-b8ca-d7a1063f5916" (UID: "12f8d03f-3f24-4a86-b8ca-d7a1063f5916"). InnerVolumeSpecName "kube-api-access-556r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:29:07.781164 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.781131 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:07.788078 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.788054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:07.793936 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.793915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" Sep 09 17:29:07.795537 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.795474 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556r9\" (UniqueName: \"kubernetes.io/projected/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-kube-api-access-556r9\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:07.795537 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.795502 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12f8d03f-3f24-4a86-b8ca-d7a1063f5916-extensions-socket-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:07.795537 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:07.795516 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz6cd\" (UniqueName: \"kubernetes.io/projected/e7d6641d-2b36-45e1-a371-625a1b730e4b-kube-api-access-kz6cd\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:08.069345 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.069311 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c"] Sep 09 17:29:08.070841 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:29:08.070808 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edc23f1_c7c2_448e_a6e6_65717671535b.slice/crio-ed3b603a68485f031f3da8db6b5c0c05329bfb8afabbffbcddb58d53850b9440 WatchSource:0}: Error finding container ed3b603a68485f031f3da8db6b5c0c05329bfb8afabbffbcddb58d53850b9440: Status 404 returned error can't find the container with id ed3b603a68485f031f3da8db6b5c0c05329bfb8afabbffbcddb58d53850b9440 Sep 09 17:29:08.072952 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.072929 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz"] Sep 09 17:29:08.074687 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:29:08.074662 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e33c3c0_24e7_4180_8029_03dd7c40f9d8.slice/crio-90d43319b7039ce672ba8392feae3b76e5ea2460bf9487237ab83e6d9108c463 WatchSource:0}: Error finding container 90d43319b7039ce672ba8392feae3b76e5ea2460bf9487237ab83e6d9108c463: Status 404 returned error can't find the container with id 90d43319b7039ce672ba8392feae3b76e5ea2460bf9487237ab83e6d9108c463 Sep 09 17:29:08.086488 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.086469 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96"] Sep 09 17:29:08.090517 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:29:08.090495 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b29873e_daf4_4b8c_93a6_77a0b0464522.slice/crio-9389d9bcdc972210f8a2b92eb52af6947877f665482bdad85781b50c36b8b7fb WatchSource:0}: Error finding container 9389d9bcdc972210f8a2b92eb52af6947877f665482bdad85781b50c36b8b7fb: Status 404 returned error can't find the container with id 9389d9bcdc972210f8a2b92eb52af6947877f665482bdad85781b50c36b8b7fb Sep 09 17:29:08.098988 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.098965 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vpb4q" Sep 09 17:29:08.099083 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.099032 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-qcjwm" Sep 09 17:29:08.101261 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:08.101234 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5dgxx\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:08.103410 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:08.103380 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-568hr\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:08.132669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.132641 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" path="/var/lib/kubelet/pods/12f8d03f-3f24-4a86-b8ca-d7a1063f5916/volumes" Sep 09 17:29:08.133115 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.133098 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d6641d-2b36-45e1-a371-625a1b730e4b" path="/var/lib/kubelet/pods/e7d6641d-2b36-45e1-a371-625a1b730e4b/volumes" Sep 09 17:29:08.164146 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.164124 2580 generic.go:356] "Generic (PLEG): container finished" podID="12f8d03f-3f24-4a86-b8ca-d7a1063f5916" containerID="c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b" exitCode=0 Sep 09 17:29:08.164259 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.164167 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-568hr" Sep 09 17:29:08.164259 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.164201 2580 scope.go:117] "RemoveContainer" containerID="c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b" Sep 09 17:29:08.166349 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:08.166328 2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" container="manager" Sep 09 17:29:08.167910 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.167884 2580 generic.go:356] "Generic (PLEG): container finished" podID="e7d6641d-2b36-45e1-a371-625a1b730e4b" containerID="19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e" exitCode=0 Sep 09 17:29:08.168028 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.167921 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5dgxx" Sep 09 17:29:08.170160 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.170138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" event={"ID":"1e33c3c0-24e7-4180-8029-03dd7c40f9d8","Type":"ContainerStarted","Data":"efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1"} Sep 09 17:29:08.170416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.170331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" event={"ID":"1e33c3c0-24e7-4180-8029-03dd7c40f9d8","Type":"ContainerStarted","Data":"90d43319b7039ce672ba8392feae3b76e5ea2460bf9487237ab83e6d9108c463"} Sep 09 17:29:08.170416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.170386 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:08.173138 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.173100 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" event={"ID":"3edc23f1-c7c2-448e-a6e6-65717671535b","Type":"ContainerStarted","Data":"cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28"} Sep 09 17:29:08.173252 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.173188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" event={"ID":"3edc23f1-c7c2-448e-a6e6-65717671535b","Type":"ContainerStarted","Data":"ed3b603a68485f031f3da8db6b5c0c05329bfb8afabbffbcddb58d53850b9440"} Sep 09 17:29:08.173334 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.173319 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:08.175271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.175245 2580 scope.go:117] "RemoveContainer" containerID="c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b" Sep 09 17:29:08.175744 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:08.175692 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b\": container with ID starting with c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b not found: ID does not exist" containerID="c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b" Sep 09 17:29:08.175879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.175767 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b"} err="failed to get container status \"c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b\": rpc error: code = NotFound desc = could not find container \"c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b\": container with ID starting with c9c74e39a75be523e8837363d7e853e2c05858f6dd54241700ed6f4805003e3b not found: ID does not exist" Sep 09 17:29:08.175879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.175790 2580 scope.go:117] "RemoveContainer" containerID="19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e" Sep 09 17:29:08.176522 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.176484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" event={"ID":"9b29873e-daf4-4b8c-93a6-77a0b0464522","Type":"ContainerStarted","Data":"bfeb6f509733afed3dc0c6019486b5d6e2d389cfffe37c15676f9da5c635a612"} Sep 09 17:29:08.176609 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.176525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" event={"ID":"9b29873e-daf4-4b8c-93a6-77a0b0464522","Type":"ContainerStarted","Data":"9389d9bcdc972210f8a2b92eb52af6947877f665482bdad85781b50c36b8b7fb"} Sep 09 17:29:08.179748 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.179732 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" Sep 09 17:29:08.185830 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.185706 2580 scope.go:117] "RemoveContainer" containerID="19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e" Sep 09 17:29:08.186183 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:08.186157 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e\": container with ID starting with 19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e not found: ID does not exist" containerID="19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e" Sep 09 17:29:08.186261 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.186194 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e"} err="failed to get container status \"19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e\": rpc error: code = NotFound desc = could not find container \"19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e\": container with ID starting with 19be9600ba60aa5012fd0dcbb378761564ff125f1ea75a487626c118d52e321e not found: ID does not exist" Sep 09 17:29:08.226358 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.226309 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" podStartSLOduration=1.226293943 podStartE2EDuration="1.226293943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:08.225568489 +0000 UTC m=+742.573929962" watchObservedRunningTime="2026-09-09 17:29:08.226293943 +0000 UTC m=+742.574655415" Sep 09 17:29:08.262145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:08.262098 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" podStartSLOduration=1.262083149 podStartE2EDuration="1.262083149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:08.258979004 +0000 UTC m=+742.607340498" watchObservedRunningTime="2026-09-09 17:29:08.262083149 +0000 UTC m=+742.610444621" Sep 09 17:29:09.188301 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:09.188266 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" podUID="d4514f5c-da70-418b-a681-8ea4585d2392" containerName="manager" containerID="cri-o://7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e" gracePeriod=2 Sep 09 17:29:09.190591 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:09.190563 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="d4514f5c-da70-418b-a681-8ea4585d2392" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:09.432108 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:09.432086 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:09.434542 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:09.434513 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="d4514f5c-da70-418b-a681-8ea4585d2392" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:09.510766 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:09.510746 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4514f5c-da70-418b-a681-8ea4585d2392-extensions-socket-volume\") pod \"d4514f5c-da70-418b-a681-8ea4585d2392\" (UID: \"d4514f5c-da70-418b-a681-8ea4585d2392\") " Sep 09 17:29:09.510858 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:09.510799 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqgqn\" (UniqueName: \"kubernetes.io/projected/d4514f5c-da70-418b-a681-8ea4585d2392-kube-api-access-dqgqn\") pod \"d4514f5c-da70-418b-a681-8ea4585d2392\" (UID: \"d4514f5c-da70-418b-a681-8ea4585d2392\") " Sep 09 17:29:09.510991 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:09.510968 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4514f5c-da70-418b-a681-8ea4585d2392-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d4514f5c-da70-418b-a681-8ea4585d2392" (UID: "d4514f5c-da70-418b-a681-8ea4585d2392"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:29:09.512680 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:09.512659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4514f5c-da70-418b-a681-8ea4585d2392-kube-api-access-dqgqn" (OuterVolumeSpecName: "kube-api-access-dqgqn") pod "d4514f5c-da70-418b-a681-8ea4585d2392" (UID: "d4514f5c-da70-418b-a681-8ea4585d2392"). InnerVolumeSpecName "kube-api-access-dqgqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:29:09.611600 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:09.611578 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4514f5c-da70-418b-a681-8ea4585d2392-extensions-socket-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:09.611600 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:09.611597 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqgqn\" (UniqueName: \"kubernetes.io/projected/d4514f5c-da70-418b-a681-8ea4585d2392-kube-api-access-dqgqn\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:10.131558 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:10.131523 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4514f5c-da70-418b-a681-8ea4585d2392" path="/var/lib/kubelet/pods/d4514f5c-da70-418b-a681-8ea4585d2392/volumes" Sep 09 17:29:10.195731 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:10.195684 2580 generic.go:356] "Generic (PLEG): container finished" podID="d4514f5c-da70-418b-a681-8ea4585d2392" containerID="7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e" exitCode=2 Sep 09 17:29:10.196132 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:10.195759 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:10.196132 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:10.195787 2580 scope.go:117] "RemoveContainer" containerID="7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e" Sep 09 17:29:10.200309 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:10.200282 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-mptjq\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="d4514f5c-da70-418b-a681-8ea4585d2392" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-mptjq" Sep 09 17:29:10.204095 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:10.204073 2580 scope.go:117] "RemoveContainer" containerID="7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e" Sep 09 17:29:10.204350 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:10.204331 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e\": container with ID starting with 7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e not found: ID does not exist" containerID="7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e" Sep 09 17:29:10.204428 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:10.204361 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e"} err="failed to get container status \"7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e\": rpc error: code = NotFound desc = could not find container \"7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e\": container with ID starting with 7739db642724e984bb2198c23c6a0e824b08cdc4631ff4e363f6887f1846107e not found: ID does not exist" Sep 09 17:29:19.192235 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:19.192203 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" Sep 09 17:29:19.192724 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:19.192273 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:19.233665 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:19.233620 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-swb96" podStartSLOduration=12.233606095 podStartE2EDuration="12.233606095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:08.291029331 +0000 UTC m=+742.639390805" watchObservedRunningTime="2026-09-09 17:29:19.233606095 +0000 UTC m=+753.581967567" Sep 09 17:29:20.198111 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.198081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:20.264573 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.264543 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz"] Sep 09 17:29:20.264774 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.264753 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" podUID="1e33c3c0-24e7-4180-8029-03dd7c40f9d8" containerName="manager" containerID="cri-o://efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1" gracePeriod=10 Sep 09 17:29:20.512054 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.512030 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:20.556377 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.556342 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf"] Sep 09 17:29:20.556778 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.556764 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e33c3c0-24e7-4180-8029-03dd7c40f9d8" containerName="manager" Sep 09 17:29:20.556778 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.556779 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e33c3c0-24e7-4180-8029-03dd7c40f9d8" containerName="manager" Sep 09 17:29:20.556872 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.556864 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e33c3c0-24e7-4180-8029-03dd7c40f9d8" containerName="manager" Sep 09 17:29:20.560390 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.560367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:20.576590 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.576566 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf"] Sep 09 17:29:20.607005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.606977 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn4sx\" (UniqueName: \"kubernetes.io/projected/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-kube-api-access-zn4sx\") pod \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\" (UID: \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\") " Sep 09 17:29:20.607125 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.607010 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-extensions-socket-volume\") pod \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\" (UID: \"1e33c3c0-24e7-4180-8029-03dd7c40f9d8\") " Sep 09 17:29:20.607165 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.607144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43f02d6c-644b-4910-a844-022f00f5e841-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2blhf\" (UID: \"43f02d6c-644b-4910-a844-022f00f5e841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:20.607274 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.607251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnffc\" (UniqueName: \"kubernetes.io/projected/43f02d6c-644b-4910-a844-022f00f5e841-kube-api-access-xnffc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2blhf\" (UID: \"43f02d6c-644b-4910-a844-022f00f5e841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:20.607434 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.607416 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1e33c3c0-24e7-4180-8029-03dd7c40f9d8" (UID: "1e33c3c0-24e7-4180-8029-03dd7c40f9d8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:29:20.609160 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.609138 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-kube-api-access-zn4sx" (OuterVolumeSpecName: "kube-api-access-zn4sx") pod "1e33c3c0-24e7-4180-8029-03dd7c40f9d8" (UID: "1e33c3c0-24e7-4180-8029-03dd7c40f9d8"). InnerVolumeSpecName "kube-api-access-zn4sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:29:20.707915 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.707873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43f02d6c-644b-4910-a844-022f00f5e841-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2blhf\" (UID: \"43f02d6c-644b-4910-a844-022f00f5e841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:20.708108 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.707962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnffc\" (UniqueName: \"kubernetes.io/projected/43f02d6c-644b-4910-a844-022f00f5e841-kube-api-access-xnffc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2blhf\" (UID: \"43f02d6c-644b-4910-a844-022f00f5e841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:20.708108 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.708016 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn4sx\" (UniqueName: \"kubernetes.io/projected/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-kube-api-access-zn4sx\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:20.708108 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.708036 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e33c3c0-24e7-4180-8029-03dd7c40f9d8-extensions-socket-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:20.708273 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.708254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43f02d6c-644b-4910-a844-022f00f5e841-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2blhf\" (UID: \"43f02d6c-644b-4910-a844-022f00f5e841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:20.716278 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.716257 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnffc\" (UniqueName: \"kubernetes.io/projected/43f02d6c-644b-4910-a844-022f00f5e841-kube-api-access-xnffc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2blhf\" (UID: \"43f02d6c-644b-4910-a844-022f00f5e841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:20.870440 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:20.870404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:21.146990 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.146966 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf"] Sep 09 17:29:21.148979 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:29:21.148954 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f02d6c_644b_4910_a844_022f00f5e841.slice/crio-7dfc10644035aa8e1b17a4ebba1dd46620eed58a3ade83cb8d851440ce483236 WatchSource:0}: Error finding container 7dfc10644035aa8e1b17a4ebba1dd46620eed58a3ade83cb8d851440ce483236: Status 404 returned error can't find the container with id 7dfc10644035aa8e1b17a4ebba1dd46620eed58a3ade83cb8d851440ce483236 Sep 09 17:29:21.256675 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.256640 2580 generic.go:356] "Generic (PLEG): container finished" podID="1e33c3c0-24e7-4180-8029-03dd7c40f9d8" containerID="efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1" exitCode=0 Sep 09 17:29:21.257137 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.256679 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" Sep 09 17:29:21.257137 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.256727 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" event={"ID":"1e33c3c0-24e7-4180-8029-03dd7c40f9d8","Type":"ContainerDied","Data":"efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1"} Sep 09 17:29:21.257137 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.256777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz" event={"ID":"1e33c3c0-24e7-4180-8029-03dd7c40f9d8","Type":"ContainerDied","Data":"90d43319b7039ce672ba8392feae3b76e5ea2460bf9487237ab83e6d9108c463"} Sep 09 17:29:21.257137 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.256817 2580 scope.go:117] "RemoveContainer" containerID="efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1" Sep 09 17:29:21.258683 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.258658 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" event={"ID":"43f02d6c-644b-4910-a844-022f00f5e841","Type":"ContainerStarted","Data":"2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87"} Sep 09 17:29:21.258884 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.258706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" event={"ID":"43f02d6c-644b-4910-a844-022f00f5e841","Type":"ContainerStarted","Data":"7dfc10644035aa8e1b17a4ebba1dd46620eed58a3ade83cb8d851440ce483236"} Sep 09 17:29:21.258884 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.258777 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:21.266081 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.265927 2580 scope.go:117] "RemoveContainer" containerID="efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1" Sep 09 17:29:21.266210 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:21.266192 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1\": container with ID starting with efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1 not found: ID does not exist" containerID="efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1" Sep 09 17:29:21.266253 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.266219 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1"} err="failed to get container status \"efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1\": rpc error: code = NotFound desc = could not find container \"efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1\": container with ID starting with efc18f233403547192967fc322580409bf6229ff92f04ef4659a07120fb511c1 not found: ID does not exist" Sep 09 17:29:21.280061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.280010 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" podStartSLOduration=1.279998248 podStartE2EDuration="1.279998248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:21.278645343 +0000 UTC m=+755.627006814" watchObservedRunningTime="2026-09-09 17:29:21.279998248 +0000 UTC m=+755.628359721" Sep 09 17:29:21.293019 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.292998 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz"] Sep 09 17:29:21.296821 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:21.296798 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-42cxz"] Sep 09 17:29:22.136888 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:22.136856 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e33c3c0-24e7-4180-8029-03dd7c40f9d8" path="/var/lib/kubelet/pods/1e33c3c0-24e7-4180-8029-03dd7c40f9d8/volumes" Sep 09 17:29:32.267512 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.267472 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:29:32.313399 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.313362 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c"] Sep 09 17:29:32.313670 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.313620 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" podUID="3edc23f1-c7c2-448e-a6e6-65717671535b" containerName="manager" containerID="cri-o://cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28" gracePeriod=10 Sep 09 17:29:32.561038 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.561016 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:32.717416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.717381 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q29m\" (UniqueName: \"kubernetes.io/projected/3edc23f1-c7c2-448e-a6e6-65717671535b-kube-api-access-7q29m\") pod \"3edc23f1-c7c2-448e-a6e6-65717671535b\" (UID: \"3edc23f1-c7c2-448e-a6e6-65717671535b\") " Sep 09 17:29:32.717583 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.717435 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3edc23f1-c7c2-448e-a6e6-65717671535b-extensions-socket-volume\") pod \"3edc23f1-c7c2-448e-a6e6-65717671535b\" (UID: \"3edc23f1-c7c2-448e-a6e6-65717671535b\") " Sep 09 17:29:32.717815 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.717787 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edc23f1-c7c2-448e-a6e6-65717671535b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3edc23f1-c7c2-448e-a6e6-65717671535b" (UID: "3edc23f1-c7c2-448e-a6e6-65717671535b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:29:32.719570 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.719547 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edc23f1-c7c2-448e-a6e6-65717671535b-kube-api-access-7q29m" (OuterVolumeSpecName: "kube-api-access-7q29m") pod "3edc23f1-c7c2-448e-a6e6-65717671535b" (UID: "3edc23f1-c7c2-448e-a6e6-65717671535b"). InnerVolumeSpecName "kube-api-access-7q29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:29:32.818930 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.818858 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q29m\" (UniqueName: \"kubernetes.io/projected/3edc23f1-c7c2-448e-a6e6-65717671535b-kube-api-access-7q29m\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:32.818930 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:32.818883 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3edc23f1-c7c2-448e-a6e6-65717671535b-extensions-socket-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:29:33.338941 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.338905 2580 generic.go:356] "Generic (PLEG): container finished" podID="3edc23f1-c7c2-448e-a6e6-65717671535b" containerID="cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28" exitCode=0 Sep 09 17:29:33.338941 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.338944 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" event={"ID":"3edc23f1-c7c2-448e-a6e6-65717671535b","Type":"ContainerDied","Data":"cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28"} Sep 09 17:29:33.339428 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.338981 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" event={"ID":"3edc23f1-c7c2-448e-a6e6-65717671535b","Type":"ContainerDied","Data":"ed3b603a68485f031f3da8db6b5c0c05329bfb8afabbffbcddb58d53850b9440"} Sep 09 17:29:33.339428 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.339003 2580 scope.go:117] "RemoveContainer" containerID="cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28" Sep 09 17:29:33.339428 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.338946 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c" Sep 09 17:29:33.348198 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.348182 2580 scope.go:117] "RemoveContainer" containerID="cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28" Sep 09 17:29:33.348445 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:29:33.348427 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28\": container with ID starting with cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28 not found: ID does not exist" containerID="cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28" Sep 09 17:29:33.348499 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.348453 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28"} err="failed to get container status \"cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28\": rpc error: code = NotFound desc = could not find container \"cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28\": container with ID starting with cca86efa4952b0f38ebbf3538109df81698600b599b8b9c334a27309c7a38d28 not found: ID does not exist" Sep 09 17:29:33.361439 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.361417 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c"] Sep 09 17:29:33.366619 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:33.366599 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-psz2c"] Sep 09 17:29:34.131318 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:34.131276 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3edc23f1-c7c2-448e-a6e6-65717671535b" path="/var/lib/kubelet/pods/3edc23f1-c7c2-448e-a6e6-65717671535b/volumes" Sep 09 17:29:36.590280 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.590251 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4"] Sep 09 17:29:36.590853 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.590835 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3edc23f1-c7c2-448e-a6e6-65717671535b" containerName="manager" Sep 09 17:29:36.590941 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.590855 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3edc23f1-c7c2-448e-a6e6-65717671535b" containerName="manager" Sep 09 17:29:36.590990 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.590948 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3edc23f1-c7c2-448e-a6e6-65717671535b" containerName="manager" Sep 09 17:29:36.594323 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.594302 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.596917 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.596874 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 17:29:36.596917 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.596888 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6nsqw\"" Sep 09 17:29:36.603584 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.603562 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4"] Sep 09 17:29:36.751944 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.751912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.752103 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.751951 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmpx\" (UniqueName: \"kubernetes.io/projected/e078a357-0326-49bd-901c-31cbcaad0861-kube-api-access-lrmpx\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.752103 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.752028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.752103 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.752054 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.752214 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.752113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.752214 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.752143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e078a357-0326-49bd-901c-31cbcaad0861-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.752214 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.752172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e078a357-0326-49bd-901c-31cbcaad0861-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.752214 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.752198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.752338 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.752260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e078a357-0326-49bd-901c-31cbcaad0861-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.852950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.852990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e078a357-0326-49bd-901c-31cbcaad0861-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853275 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmpx\" (UniqueName: \"kubernetes.io/projected/e078a357-0326-49bd-901c-31cbcaad0861-kube-api-access-lrmpx\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853275 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853275 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853275 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853275 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e078a357-0326-49bd-901c-31cbcaad0861-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853275 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853202 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e078a357-0326-49bd-901c-31cbcaad0861-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853572 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853386 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853670 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853789 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853789 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853766 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.853932 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.853911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e078a357-0326-49bd-901c-31cbcaad0861-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.855487 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.855463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e078a357-0326-49bd-901c-31cbcaad0861-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.855487 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.855482 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e078a357-0326-49bd-901c-31cbcaad0861-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.862211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.862187 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e078a357-0326-49bd-901c-31cbcaad0861-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.862301 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.862291 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmpx\" (UniqueName: \"kubernetes.io/projected/e078a357-0326-49bd-901c-31cbcaad0861-kube-api-access-lrmpx\") pod \"maas-default-gateway-openshift-default-78bb78ff78-bwbv4\" (UID: \"e078a357-0326-49bd-901c-31cbcaad0861\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:36.905057 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:36.905031 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:37.177075 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:37.177053 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4"] Sep 09 17:29:37.178794 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:29:37.178766 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode078a357_0326_49bd_901c_31cbcaad0861.slice/crio-3842711c5079df376777abc8ea55c735bf913a1b26839ace63f7e523b990d3f4 WatchSource:0}: Error finding container 3842711c5079df376777abc8ea55c735bf913a1b26839ace63f7e523b990d3f4: Status 404 returned error can't find the container with id 3842711c5079df376777abc8ea55c735bf913a1b26839ace63f7e523b990d3f4 Sep 09 17:29:37.362387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:37.362352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" event={"ID":"e078a357-0326-49bd-901c-31cbcaad0861","Type":"ContainerStarted","Data":"3842711c5079df376777abc8ea55c735bf913a1b26839ace63f7e523b990d3f4"} Sep 09 17:29:39.677429 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:39.677388 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:29:39.677656 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:39.677494 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:29:39.677656 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:39.677523 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:29:40.387259 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:40.387228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" event={"ID":"e078a357-0326-49bd-901c-31cbcaad0861","Type":"ContainerStarted","Data":"0eebbe634bb97c527f40ad7cfd4765f3f76ba525196c59b73881b2aedc479f93"} Sep 09 17:29:40.407225 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:40.407177 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" podStartSLOduration=1.9109381939999999 podStartE2EDuration="4.407162808s" totalImagesPullingTime="2.496224614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:40.405830048 +0000 UTC m=+774.754191521" watchObservedRunningTime="2026-09-09 17:29:40.407162808 +0000 UTC m=+774.755524285" Sep 09 17:29:40.905451 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:40.905419 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:40.910232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:40.910208 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:41.392398 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:41.392361 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:41.393238 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:41.393219 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:29:49.277024 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.276989 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:29:49.282282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.282261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:49.284793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.284773 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 09 17:29:49.288524 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.288506 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:29:49.318429 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.318405 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:29:49.462101 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.462067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fxv\" (UniqueName: \"kubernetes.io/projected/14ad9c8c-3e0e-4c73-8462-8919f884220f-kube-api-access-97fxv\") pod \"limitador-limitador-84994f7ffc-tzkqv\" (UID: \"14ad9c8c-3e0e-4c73-8462-8919f884220f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:49.462294 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.462136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/14ad9c8c-3e0e-4c73-8462-8919f884220f-config-file\") pod \"limitador-limitador-84994f7ffc-tzkqv\" (UID: \"14ad9c8c-3e0e-4c73-8462-8919f884220f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:49.562683 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.562603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97fxv\" (UniqueName: \"kubernetes.io/projected/14ad9c8c-3e0e-4c73-8462-8919f884220f-kube-api-access-97fxv\") pod \"limitador-limitador-84994f7ffc-tzkqv\" (UID: \"14ad9c8c-3e0e-4c73-8462-8919f884220f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:49.562852 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.562702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/14ad9c8c-3e0e-4c73-8462-8919f884220f-config-file\") pod \"limitador-limitador-84994f7ffc-tzkqv\" (UID: \"14ad9c8c-3e0e-4c73-8462-8919f884220f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:49.563299 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.563279 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/14ad9c8c-3e0e-4c73-8462-8919f884220f-config-file\") pod \"limitador-limitador-84994f7ffc-tzkqv\" (UID: \"14ad9c8c-3e0e-4c73-8462-8919f884220f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:49.570258 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.570228 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fxv\" (UniqueName: \"kubernetes.io/projected/14ad9c8c-3e0e-4c73-8462-8919f884220f-kube-api-access-97fxv\") pod \"limitador-limitador-84994f7ffc-tzkqv\" (UID: \"14ad9c8c-3e0e-4c73-8462-8919f884220f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:49.593210 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.593183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:49.864378 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:49.864348 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:29:49.866420 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:29:49.866396 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ad9c8c_3e0e_4c73_8462_8919f884220f.slice/crio-6ec048e1652dab19216f3c75d4f2b8c3d8385a76b1b641ad8772941e55a46f74 WatchSource:0}: Error finding container 6ec048e1652dab19216f3c75d4f2b8c3d8385a76b1b641ad8772941e55a46f74: Status 404 returned error can't find the container with id 6ec048e1652dab19216f3c75d4f2b8c3d8385a76b1b641ad8772941e55a46f74 Sep 09 17:29:50.447281 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:50.447230 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" event={"ID":"14ad9c8c-3e0e-4c73-8462-8919f884220f","Type":"ContainerStarted","Data":"6ec048e1652dab19216f3c75d4f2b8c3d8385a76b1b641ad8772941e55a46f74"} Sep 09 17:29:51.453869 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:51.453834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" event={"ID":"14ad9c8c-3e0e-4c73-8462-8919f884220f","Type":"ContainerStarted","Data":"0d7ee14e2d019b822a9b5ce774455e4a4ac239df78820d448b75b3baf98e7b7f"} Sep 09 17:29:51.454267 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:51.453919 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:29:51.469879 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:29:51.469834 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" podStartSLOduration=1.5558208260000002 podStartE2EDuration="2.469820273s" totalImagesPullingTime="913.999447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:29:51.469468039 +0000 UTC m=+785.817829521" watchObservedRunningTime="2026-09-09 17:29:51.469820273 +0000 UTC m=+785.818181745" Sep 09 17:30:02.460284 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:30:02.460256 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tzkqv" Sep 09 17:31:04.609285 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:04.609255 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:31:12.433881 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:12.433842 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bwbv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 09 17:31:12.434340 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:12.433925 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" podUID="e078a357-0326-49bd-901c-31cbcaad0861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 17:31:13.898581 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:13.898546 2580 generic.go:356] "Generic (PLEG): container finished" podID="e078a357-0326-49bd-901c-31cbcaad0861" containerID="0eebbe634bb97c527f40ad7cfd4765f3f76ba525196c59b73881b2aedc479f93" exitCode=0 Sep 09 17:31:13.899071 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:13.898590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" event={"ID":"e078a357-0326-49bd-901c-31cbcaad0861","Type":"ContainerDied","Data":"0eebbe634bb97c527f40ad7cfd4765f3f76ba525196c59b73881b2aedc479f93"} Sep 09 17:31:13.899071 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:13.898969 2580 scope.go:117] "RemoveContainer" containerID="0eebbe634bb97c527f40ad7cfd4765f3f76ba525196c59b73881b2aedc479f93" Sep 09 17:31:13.899936 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:13.899911 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:31:13.899990 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:13.899978 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:31:13.900041 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:13.900011 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:31:14.905466 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:14.905435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" event={"ID":"e078a357-0326-49bd-901c-31cbcaad0861","Type":"ContainerStarted","Data":"eec6ec16fcdaf51145799d7719e33e026af72caacb1363a4f2e3b8bbcc83dcff"} Sep 09 17:31:15.905376 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:15.905335 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:31:15.905582 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:15.905561 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bwbv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 09 17:31:15.905980 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:15.905615 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" podUID="e078a357-0326-49bd-901c-31cbcaad0861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 09 17:31:16.906054 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:16.906019 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bwbv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 09 17:31:16.906443 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:16.906080 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" podUID="e078a357-0326-49bd-901c-31cbcaad0861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 09 17:31:17.906065 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:17.906030 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bwbv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 09 17:31:17.906439 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:17.906087 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" podUID="e078a357-0326-49bd-901c-31cbcaad0861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 09 17:31:18.905355 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:18.905320 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-bwbv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 09 17:31:18.905595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:18.905387 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" podUID="e078a357-0326-49bd-901c-31cbcaad0861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 09 17:31:19.908787 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:19.908754 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:31:19.909229 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:19.909015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:31:19.909588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:19.909570 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-bwbv4" Sep 09 17:31:25.989580 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:25.989545 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb"] Sep 09 17:31:25.993667 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:25.993644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:25.997468 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:25.997446 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7dhnx\"" Sep 09 17:31:25.997468 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:25.997463 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 09 17:31:25.997641 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:25.997448 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 17:31:25.997641 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:25.997480 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 17:31:26.003293 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.003264 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb"] Sep 09 17:31:26.070669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.070639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.070851 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.070677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.070851 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.070696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jf66\" (UniqueName: \"kubernetes.io/projected/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-kube-api-access-4jf66\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.070851 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.070742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.070851 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.070771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.071011 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.070845 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.171780 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.171750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.172014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.171822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.172014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.171848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.172014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.171865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jf66\" (UniqueName: \"kubernetes.io/projected/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-kube-api-access-4jf66\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.172014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.171887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.172014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.171923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.172333 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.172297 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.172414 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.172375 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.172414 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.172391 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.174098 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.174073 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.174358 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.174339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.179328 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.179307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jf66\" (UniqueName: \"kubernetes.io/projected/3edaa1de-d71a-42e4-a3ec-cd79b51a5e97-kube-api-access-4jf66\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb\" (UID: \"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.305237 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.305203 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:26.580231 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.580207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb"] Sep 09 17:31:26.582746 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:31:26.582689 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edaa1de_d71a_42e4_a3ec_cd79b51a5e97.slice/crio-62d45607e5f7e5b68473fd9b28941c97e22f552ee62a3b42ff9ee49aba422033 WatchSource:0}: Error finding container 62d45607e5f7e5b68473fd9b28941c97e22f552ee62a3b42ff9ee49aba422033: Status 404 returned error can't find the container with id 62d45607e5f7e5b68473fd9b28941c97e22f552ee62a3b42ff9ee49aba422033 Sep 09 17:31:26.606580 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.606554 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:31:26.980490 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:26.980358 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" event={"ID":"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97","Type":"ContainerStarted","Data":"62d45607e5f7e5b68473fd9b28941c97e22f552ee62a3b42ff9ee49aba422033"} Sep 09 17:31:27.989697 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:27.989662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" event={"ID":"3edaa1de-d71a-42e4-a3ec-cd79b51a5e97","Type":"ContainerStarted","Data":"c3dfbd8be1c95be4b19fc20e507868446416831406376826b0b396877dda5289"} Sep 09 17:31:28.006497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:28.006439 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" podStartSLOduration=1.846225371 podStartE2EDuration="3.00641998s" totalImagesPullingTime="1.160194609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:31:28.006325768 +0000 UTC m=+882.354687241" watchObservedRunningTime="2026-09-09 17:31:28.00641998 +0000 UTC m=+882.354781452" Sep 09 17:31:34.700130 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:34.700091 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:31:36.305567 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:36.305529 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:36.306032 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:36.305586 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:36.319880 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:36.319853 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:37.053474 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:37.053446 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb" Sep 09 17:31:41.784834 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:41.784750 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:31:46.203211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:46.203184 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:31:46.205962 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:46.205941 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:31:46.206490 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:46.206471 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:31:46.209100 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:46.209082 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:31:50.478529 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.478498 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk"] Sep 09 17:31:50.483024 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.483000 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.484385 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.484357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.484499 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.484398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.484499 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.484454 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7kx\" (UniqueName: \"kubernetes.io/projected/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-kube-api-access-zt7kx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.484499 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.484492 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.484640 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.484525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.484640 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.484541 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.485492 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.485476 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 09 17:31:50.492725 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.492680 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk"] Sep 09 17:31:50.585635 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.585602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.585635 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.585640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.585870 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.585675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7kx\" (UniqueName: \"kubernetes.io/projected/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-kube-api-access-zt7kx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.585870 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.585735 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.585870 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.585768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.585870 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.585790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.586066 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.586047 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.586158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.586135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.586230 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.586175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.587951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.587926 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.588284 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.588266 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.593327 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.593307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7kx\" (UniqueName: \"kubernetes.io/projected/8dc5fca2-0b1d-412b-bb3a-e7f54e825956-kube-api-access-zt7kx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-vhptk\" (UID: \"8dc5fca2-0b1d-412b-bb3a-e7f54e825956\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:50.795654 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:50.795615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:31:51.069887 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:51.069860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk"] Sep 09 17:31:51.071665 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:31:51.071638 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc5fca2_0b1d_412b_bb3a_e7f54e825956.slice/crio-fee217fff9fc5bf4e9ec0db48d65707158ea2e4574d42f96a47b9b19c11db9ce WatchSource:0}: Error finding container fee217fff9fc5bf4e9ec0db48d65707158ea2e4574d42f96a47b9b19c11db9ce: Status 404 returned error can't find the container with id fee217fff9fc5bf4e9ec0db48d65707158ea2e4574d42f96a47b9b19c11db9ce Sep 09 17:31:51.119357 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:51.119325 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" event={"ID":"8dc5fca2-0b1d-412b-bb3a-e7f54e825956","Type":"ContainerStarted","Data":"fee217fff9fc5bf4e9ec0db48d65707158ea2e4574d42f96a47b9b19c11db9ce"} Sep 09 17:31:51.681109 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:51.681027 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:31:52.133601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:52.133566 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" event={"ID":"8dc5fca2-0b1d-412b-bb3a-e7f54e825956","Type":"ContainerStarted","Data":"d2a1a43faecd82a493e43a7ef6ad076619eda0eea10f584a23504f82e4f80ac4"} Sep 09 17:31:52.152454 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:31:52.152405 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" podStartSLOduration=1.942998252 podStartE2EDuration="2.152389287s" totalImagesPullingTime="209.391035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:31:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:31:52.150069596 +0000 UTC m=+906.498431070" watchObservedRunningTime="2026-09-09 17:31:52.152389287 +0000 UTC m=+906.500750794" Sep 09 17:32:00.796006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:32:00.795971 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:32:00.796410 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:32:00.796071 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:32:00.808164 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:32:00.808138 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:32:01.195332 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:32:01.195252 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-vhptk" Sep 09 17:32:03.680024 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:32:03.679987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:32:14.992951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:32:14.992913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:32:23.482830 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:32:23.482789 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:33:46.212469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:33:46.212391 2580 scope.go:117] "RemoveContainer" containerID="2ba41240f3d3bdc949975a0239838e6f5065aeaa97128ba25071297d898c07cb" Sep 09 17:34:03.213596 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.213560 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7676-139909155909296"] Sep 09 17:34:03.218689 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.218668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-139909155909296" Sep 09 17:34:03.222416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.222400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p6jf9\"" Sep 09 17:34:03.238696 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.238670 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-139909155909296"] Sep 09 17:34:03.384418 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.384381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzdz\" (UniqueName: \"kubernetes.io/projected/aa9e74ee-d443-45f7-a0af-5f17ce1acfcb-kube-api-access-vhzdz\") pod \"test-curl-7676-139909155909296\" (UID: \"aa9e74ee-d443-45f7-a0af-5f17ce1acfcb\") " pod="opendatahub/test-curl-7676-139909155909296" Sep 09 17:34:03.485784 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.485668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzdz\" (UniqueName: \"kubernetes.io/projected/aa9e74ee-d443-45f7-a0af-5f17ce1acfcb-kube-api-access-vhzdz\") pod \"test-curl-7676-139909155909296\" (UID: \"aa9e74ee-d443-45f7-a0af-5f17ce1acfcb\") " pod="opendatahub/test-curl-7676-139909155909296" Sep 09 17:34:03.494241 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.494203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzdz\" (UniqueName: \"kubernetes.io/projected/aa9e74ee-d443-45f7-a0af-5f17ce1acfcb-kube-api-access-vhzdz\") pod \"test-curl-7676-139909155909296\" (UID: \"aa9e74ee-d443-45f7-a0af-5f17ce1acfcb\") " pod="opendatahub/test-curl-7676-139909155909296" Sep 09 17:34:03.527849 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.527816 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-139909155909296" Sep 09 17:34:03.797536 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.797511 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-139909155909296"] Sep 09 17:34:03.799739 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:03.799688 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa9e74ee_d443_45f7_a0af_5f17ce1acfcb.slice/crio-eae174cb209b58a8c67618159fbdf8cc28e8973001fa456eb8917105fde60116 WatchSource:0}: Error finding container eae174cb209b58a8c67618159fbdf8cc28e8973001fa456eb8917105fde60116: Status 404 returned error can't find the container with id eae174cb209b58a8c67618159fbdf8cc28e8973001fa456eb8917105fde60116 Sep 09 17:34:03.801031 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.801014 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:34:03.862429 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:03.862399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-139909155909296" event={"ID":"aa9e74ee-d443-45f7-a0af-5f17ce1acfcb","Type":"ContainerStarted","Data":"eae174cb209b58a8c67618159fbdf8cc28e8973001fa456eb8917105fde60116"} Sep 09 17:34:04.893394 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:04.893360 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:34:06.883760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:06.883702 2580 generic.go:356] "Generic (PLEG): container finished" podID="aa9e74ee-d443-45f7-a0af-5f17ce1acfcb" containerID="cd2459227f00da3f2612b69c168e8a3ca25b41b990f190cedd81f466260577e3" exitCode=0 Sep 09 17:34:06.884194 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:06.883803 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-139909155909296" event={"ID":"aa9e74ee-d443-45f7-a0af-5f17ce1acfcb","Type":"ContainerDied","Data":"cd2459227f00da3f2612b69c168e8a3ca25b41b990f190cedd81f466260577e3"} Sep 09 17:34:06.989049 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:06.989020 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d"] Sep 09 17:34:06.992990 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:06.992961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:06.995197 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:06.995172 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ea996c\"" Sep 09 17:34:06.995296 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:06.995248 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ea996c-dockercfg-9l2wb\"" Sep 09 17:34:07.002877 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.002857 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d"] Sep 09 17:34:07.119512 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.119438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d\" (UID: \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:07.119642 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.119527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67k8m\" (UniqueName: \"kubernetes.io/projected/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-kube-api-access-67k8m\") pod \"payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d\" (UID: \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:07.220844 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.220818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67k8m\" (UniqueName: \"kubernetes.io/projected/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-kube-api-access-67k8m\") pod \"payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d\" (UID: \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:07.220979 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.220878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d\" (UID: \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:07.221531 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.221511 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d\" (UID: \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:07.228244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.228221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67k8m\" (UniqueName: \"kubernetes.io/projected/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-kube-api-access-67k8m\") pod \"payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d\" (UID: \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:07.304072 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.304046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:07.574817 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.574789 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d"] Sep 09 17:34:07.575737 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:07.575676 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f44b3cc_b013_4b2b_b0fd_0aa69b7ad9f3.slice/crio-1b1710692eafd466ee945a50b2cf0e5bd5db0925ab91ef9351ea23fed6f908a0 WatchSource:0}: Error finding container 1b1710692eafd466ee945a50b2cf0e5bd5db0925ab91ef9351ea23fed6f908a0: Status 404 returned error can't find the container with id 1b1710692eafd466ee945a50b2cf0e5bd5db0925ab91ef9351ea23fed6f908a0 Sep 09 17:34:07.890295 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:07.890216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" event={"ID":"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3","Type":"ContainerStarted","Data":"1b1710692eafd466ee945a50b2cf0e5bd5db0925ab91ef9351ea23fed6f908a0"} Sep 09 17:34:08.027432 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.027404 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-139909155909296" Sep 09 17:34:08.057134 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.057110 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7676-139909155909296_aa9e74ee-d443-45f7-a0af-5f17ce1acfcb/test-curl-7676-139909155909296/0.log" Sep 09 17:34:08.080229 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.080203 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7676-139909155909296"] Sep 09 17:34:08.082081 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.082058 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7676-139909155909296"] Sep 09 17:34:08.128011 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.127985 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhzdz\" (UniqueName: \"kubernetes.io/projected/aa9e74ee-d443-45f7-a0af-5f17ce1acfcb-kube-api-access-vhzdz\") pod \"aa9e74ee-d443-45f7-a0af-5f17ce1acfcb\" (UID: \"aa9e74ee-d443-45f7-a0af-5f17ce1acfcb\") " Sep 09 17:34:08.130387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.130359 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9e74ee-d443-45f7-a0af-5f17ce1acfcb-kube-api-access-vhzdz" (OuterVolumeSpecName: "kube-api-access-vhzdz") pod "aa9e74ee-d443-45f7-a0af-5f17ce1acfcb" (UID: "aa9e74ee-d443-45f7-a0af-5f17ce1acfcb"). InnerVolumeSpecName "kube-api-access-vhzdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:34:08.131072 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.131050 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9e74ee-d443-45f7-a0af-5f17ce1acfcb" path="/var/lib/kubelet/pods/aa9e74ee-d443-45f7-a0af-5f17ce1acfcb/volumes" Sep 09 17:34:08.196116 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.196043 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7676-139909155906096"] Sep 09 17:34:08.196482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.196470 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9e74ee-d443-45f7-a0af-5f17ce1acfcb" containerName="test-curl-7676-139909155909296" Sep 09 17:34:08.196524 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.196483 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9e74ee-d443-45f7-a0af-5f17ce1acfcb" containerName="test-curl-7676-139909155909296" Sep 09 17:34:08.196577 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.196568 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9e74ee-d443-45f7-a0af-5f17ce1acfcb" containerName="test-curl-7676-139909155909296" Sep 09 17:34:08.199666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.199651 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:08.204945 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.204920 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-139909155906096"] Sep 09 17:34:08.229506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.229476 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhzdz\" (UniqueName: \"kubernetes.io/projected/aa9e74ee-d443-45f7-a0af-5f17ce1acfcb-kube-api-access-vhzdz\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:34:08.330796 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.330763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kpld\" (UniqueName: \"kubernetes.io/projected/8f9c1242-b580-43b4-8d1f-d50fe296ab69-kube-api-access-5kpld\") pod \"test-curl-7676-139909155906096\" (UID: \"8f9c1242-b580-43b4-8d1f-d50fe296ab69\") " pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:08.432187 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.432149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kpld\" (UniqueName: \"kubernetes.io/projected/8f9c1242-b580-43b4-8d1f-d50fe296ab69-kube-api-access-5kpld\") pod \"test-curl-7676-139909155906096\" (UID: \"8f9c1242-b580-43b4-8d1f-d50fe296ab69\") " pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:08.439680 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.439645 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kpld\" (UniqueName: \"kubernetes.io/projected/8f9c1242-b580-43b4-8d1f-d50fe296ab69-kube-api-access-5kpld\") pod \"test-curl-7676-139909155906096\" (UID: \"8f9c1242-b580-43b4-8d1f-d50fe296ab69\") " pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:08.510493 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.510465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:08.792453 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.792344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7676-139909155906096"] Sep 09 17:34:08.796958 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:08.796901 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f9c1242_b580_43b4_8d1f_d50fe296ab69.slice/crio-21127af4d74609a73dc5aab1aa5f4161f013aa62f2ff3db29729d497d84250a3 WatchSource:0}: Error finding container 21127af4d74609a73dc5aab1aa5f4161f013aa62f2ff3db29729d497d84250a3: Status 404 returned error can't find the container with id 21127af4d74609a73dc5aab1aa5f4161f013aa62f2ff3db29729d497d84250a3 Sep 09 17:34:08.899119 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.899077 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-139909155909296" Sep 09 17:34:08.899511 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.899132 2580 scope.go:117] "RemoveContainer" containerID="cd2459227f00da3f2612b69c168e8a3ca25b41b990f190cedd81f466260577e3" Sep 09 17:34:08.901702 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:08.901668 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-139909155906096" event={"ID":"8f9c1242-b580-43b4-8d1f-d50fe296ab69","Type":"ContainerStarted","Data":"21127af4d74609a73dc5aab1aa5f4161f013aa62f2ff3db29729d497d84250a3"} Sep 09 17:34:09.909672 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:09.909589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" event={"ID":"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3","Type":"ContainerStarted","Data":"720e87f60d1ba6b8bb81b0db254dd14b6fe6279857c924d644937c21c6e38241"} Sep 09 17:34:09.910133 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:09.909761 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:09.911221 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:09.911196 2580 generic.go:356] "Generic (PLEG): container finished" podID="8f9c1242-b580-43b4-8d1f-d50fe296ab69" containerID="619cbede42f3e7858040c982ad0dd0f37b89ade667c45b80b0e4aae9deb4b51c" exitCode=0 Sep 09 17:34:09.911364 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:09.911295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7676-139909155906096" event={"ID":"8f9c1242-b580-43b4-8d1f-d50fe296ab69","Type":"ContainerDied","Data":"619cbede42f3e7858040c982ad0dd0f37b89ade667c45b80b0e4aae9deb4b51c"} Sep 09 17:34:09.926891 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:09.926851 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" podStartSLOduration=1.988544616 podStartE2EDuration="3.926837012s" totalImagesPullingTime="1.938292396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:09.926351005 +0000 UTC m=+1044.274712480" watchObservedRunningTime="2026-09-09 17:34:09.926837012 +0000 UTC m=+1044.275198484" Sep 09 17:34:11.056782 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.056755 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:11.058825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.058801 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kpld\" (UniqueName: \"kubernetes.io/projected/8f9c1242-b580-43b4-8d1f-d50fe296ab69-kube-api-access-5kpld\") pod \"8f9c1242-b580-43b4-8d1f-d50fe296ab69\" (UID: \"8f9c1242-b580-43b4-8d1f-d50fe296ab69\") " Sep 09 17:34:11.060843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.060819 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9c1242-b580-43b4-8d1f-d50fe296ab69-kube-api-access-5kpld" (OuterVolumeSpecName: "kube-api-access-5kpld") pod "8f9c1242-b580-43b4-8d1f-d50fe296ab69" (UID: "8f9c1242-b580-43b4-8d1f-d50fe296ab69"). InnerVolumeSpecName "kube-api-access-5kpld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:34:11.074565 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.074542 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7676-139909155906096_8f9c1242-b580-43b4-8d1f-d50fe296ab69/test-curl-7676-139909155906096/0.log" Sep 09 17:34:11.099449 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.099421 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7676-139909155906096"] Sep 09 17:34:11.103824 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.103802 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7676-139909155906096"] Sep 09 17:34:11.159776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.159743 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kpld\" (UniqueName: \"kubernetes.io/projected/8f9c1242-b580-43b4-8d1f-d50fe296ab69-kube-api-access-5kpld\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:34:11.924695 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.924667 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:11.924695 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.924687 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21127af4d74609a73dc5aab1aa5f4161f013aa62f2ff3db29729d497d84250a3" Sep 09 17:34:11.933745 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.933699 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7"] Sep 09 17:34:11.934491 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.934469 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9c1242-b580-43b4-8d1f-d50fe296ab69" containerName="test-curl-7676-139909155906096" Sep 09 17:34:11.934491 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.934491 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9c1242-b580-43b4-8d1f-d50fe296ab69" containerName="test-curl-7676-139909155906096" Sep 09 17:34:11.934679 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.934606 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9c1242-b580-43b4-8d1f-d50fe296ab69" containerName="test-curl-7676-139909155906096" Sep 09 17:34:11.938121 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.938092 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.939341 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:34:11.939316 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7676-139909155906096\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="8f9c1242-b580-43b4-8d1f-d50fe296ab69" pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:11.940483 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.940456 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-37448c-openshift-default-dockercfg-vvl8s\"" Sep 09 17:34:11.950863 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.950839 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7"] Sep 09 17:34:11.964546 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:34:11.964508 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7676-139909155906096\" is forbidden: User \"system:node:ip-10-0-128-235.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-235.ec2.internal' and this object" podUID="8f9c1242-b580-43b4-8d1f-d50fe296ab69" pod="opendatahub/test-curl-7676-139909155906096" Sep 09 17:34:11.965584 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965565 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-podinfo\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.965655 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-certs\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.965655 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948dq\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-kube-api-access-948dq\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.965655 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-token\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.965793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istiod-ca-cert\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.965793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-credential-socket\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.965793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-socket\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.965897 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-envoy\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:11.965897 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:11.965856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-data\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067076 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948dq\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-kube-api-access-948dq\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-token\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istiod-ca-cert\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-credential-socket\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-socket\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-envoy\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-data\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-podinfo\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-certs\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067953 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-credential-socket\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067953 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067686 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-socket\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067953 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067772 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-data\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.067953 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.067863 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-certs\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.068204 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.068088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istiod-ca-cert\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.069455 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.069431 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-envoy\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.069776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.069758 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-podinfo\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.075311 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.075285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-token\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.075392 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.075325 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-948dq\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-kube-api-access-948dq\") pod \"e2e-autores-37448c-openshift-default-696795cd84-xngk7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.131899 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.131857 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9c1242-b580-43b4-8d1f-d50fe296ab69" path="/var/lib/kubelet/pods/8f9c1242-b580-43b4-8d1f-d50fe296ab69/volumes" Sep 09 17:34:12.250646 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.250563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:12.529602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.529573 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7"] Sep 09 17:34:12.530998 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:12.530970 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25328c87_cc8f_40c7_ab6b_1ab8640fd0a7.slice/crio-bb664888c870dd35e2ec8350c3253ca1716ab0737acfa6e29fad2c1d3468e562 WatchSource:0}: Error finding container bb664888c870dd35e2ec8350c3253ca1716ab0737acfa6e29fad2c1d3468e562: Status 404 returned error can't find the container with id bb664888c870dd35e2ec8350c3253ca1716ab0737acfa6e29fad2c1d3468e562 Sep 09 17:34:12.533957 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.533914 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:12.534061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.534008 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:12.534061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.534055 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:12.935845 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.935689 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" event={"ID":"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7","Type":"ContainerStarted","Data":"720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d"} Sep 09 17:34:12.935845 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.935768 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" event={"ID":"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7","Type":"ContainerStarted","Data":"bb664888c870dd35e2ec8350c3253ca1716ab0737acfa6e29fad2c1d3468e562"} Sep 09 17:34:12.955314 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:12.955264 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podStartSLOduration=1.955246617 podStartE2EDuration="1.955246617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:12.954930719 +0000 UTC m=+1047.303292204" watchObservedRunningTime="2026-09-09 17:34:12.955246617 +0000 UTC m=+1047.303608089" Sep 09 17:34:13.251269 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:13.251179 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:13.256667 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:13.256640 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:13.942452 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:13.942424 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:13.943293 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:13.943274 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:14.185809 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.185775 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft"] Sep 09 17:34:14.189383 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.189367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.191868 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.191838 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-399f043b-dockercfg-mjs6h\"" Sep 09 17:34:14.191868 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.191859 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-399f043b\"" Sep 09 17:34:14.199234 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.199207 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft"] Sep 09 17:34:14.287934 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.287901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st5pf\" (UniqueName: \"kubernetes.io/projected/4811475c-3148-40f3-92dc-41223183ada6-kube-api-access-st5pf\") pod \"payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft\" (UID: \"4811475c-3148-40f3-92dc-41223183ada6\") " pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.288309 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.288026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4811475c-3148-40f3-92dc-41223183ada6-plugins-config-volume\") pod \"payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft\" (UID: \"4811475c-3148-40f3-92dc-41223183ada6\") " pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.389296 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.389260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4811475c-3148-40f3-92dc-41223183ada6-plugins-config-volume\") pod \"payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft\" (UID: \"4811475c-3148-40f3-92dc-41223183ada6\") " pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.389484 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.389352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st5pf\" (UniqueName: \"kubernetes.io/projected/4811475c-3148-40f3-92dc-41223183ada6-kube-api-access-st5pf\") pod \"payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft\" (UID: \"4811475c-3148-40f3-92dc-41223183ada6\") " pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.390020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.389995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4811475c-3148-40f3-92dc-41223183ada6-plugins-config-volume\") pod \"payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft\" (UID: \"4811475c-3148-40f3-92dc-41223183ada6\") " pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.397382 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.397362 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st5pf\" (UniqueName: \"kubernetes.io/projected/4811475c-3148-40f3-92dc-41223183ada6-kube-api-access-st5pf\") pod \"payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft\" (UID: \"4811475c-3148-40f3-92dc-41223183ada6\") " pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.501621 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.501557 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.777370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.777341 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft"] Sep 09 17:34:14.779237 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:14.779211 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4811475c_3148_40f3_92dc_41223183ada6.slice/crio-0be2e0b16d8a6d3cffcd437b28df6d92923e223ec9a0361a523e46196323e3e4 WatchSource:0}: Error finding container 0be2e0b16d8a6d3cffcd437b28df6d92923e223ec9a0361a523e46196323e3e4: Status 404 returned error can't find the container with id 0be2e0b16d8a6d3cffcd437b28df6d92923e223ec9a0361a523e46196323e3e4 Sep 09 17:34:14.950628 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.950593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" event={"ID":"4811475c-3148-40f3-92dc-41223183ada6","Type":"ContainerStarted","Data":"96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56"} Sep 09 17:34:14.950824 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.950639 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" event={"ID":"4811475c-3148-40f3-92dc-41223183ada6","Type":"ContainerStarted","Data":"0be2e0b16d8a6d3cffcd437b28df6d92923e223ec9a0361a523e46196323e3e4"} Sep 09 17:34:14.950893 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.950836 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:14.983096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:14.983022 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" podStartSLOduration=0.983007712 podStartE2EDuration="983.007712ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:14.981572267 +0000 UTC m=+1049.329933742" watchObservedRunningTime="2026-09-09 17:34:14.983007712 +0000 UTC m=+1049.331369184" Sep 09 17:34:19.495413 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:19.495379 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:34:20.918231 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:20.918200 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:34:25.959951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:25.959877 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:28.944784 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:28.944666 2580 patch_prober.go:28] interesting pod/e2e-autores-37448c-openshift-default-696795cd84-xngk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 17:34:28.944784 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:28.944760 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 17:34:29.031233 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:29.031197 2580 generic.go:356] "Generic (PLEG): container finished" podID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerID="720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d" exitCode=0 Sep 09 17:34:29.031424 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:29.031269 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" event={"ID":"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7","Type":"ContainerDied","Data":"720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d"} Sep 09 17:34:29.031674 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:29.031658 2580 scope.go:117] "RemoveContainer" containerID="720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d" Sep 09 17:34:29.032839 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:29.032808 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:29.032933 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:29.032889 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:29.032933 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:29.032925 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:30.043739 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.043591 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" event={"ID":"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7","Type":"ContainerStarted","Data":"41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915"} Sep 09 17:34:30.251969 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.251925 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:30.252182 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.252065 2580 patch_prober.go:28] interesting pod/e2e-autores-37448c-openshift-default-696795cd84-xngk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 17:34:30.252182 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.252111 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 17:34:30.504215 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.504185 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn"] Sep 09 17:34:30.511164 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.511146 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.513965 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.513914 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-53da12-openshift-default-dockercfg-vhcbt\"" Sep 09 17:34:30.517352 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.517326 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn"] Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-455z7\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-kube-api-access-455z7\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655316 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-envoy\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-token\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-data\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-certs\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655517 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-socket\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-credential-socket\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-podinfo\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.655736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.655611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istiod-ca-cert\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.757066 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.756975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-certs\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.757746 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.757650 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-certs\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.758016 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.757995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-socket\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.758460 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.758434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-credential-socket\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.758838 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.758813 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-podinfo\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.758930 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.758852 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istiod-ca-cert\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.758930 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.758917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-455z7\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-kube-api-access-455z7\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.759045 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.758949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-envoy\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.759045 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.758983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-token\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.759045 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.759014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-data\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.759205 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.758368 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-socket\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.759269 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.758817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-credential-socket\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.759368 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.759347 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-data\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.759981 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.759952 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istiod-ca-cert\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.761868 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.761837 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-envoy\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.762277 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.762255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-podinfo\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.767155 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.767135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-455z7\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-kube-api-access-455z7\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.767233 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.767205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-token\") pod \"e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:30.823894 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:30.823871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:31.121808 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:31.121767 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn"] Sep 09 17:34:31.123375 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:31.123348 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e21e9a6_f10d_4cae_984c_4c681b6b0452.slice/crio-0d53ef33222c9f078627d104e75d58267d29ff8b029fd9a36e6794ebd2e745d0 WatchSource:0}: Error finding container 0d53ef33222c9f078627d104e75d58267d29ff8b029fd9a36e6794ebd2e745d0: Status 404 returned error can't find the container with id 0d53ef33222c9f078627d104e75d58267d29ff8b029fd9a36e6794ebd2e745d0 Sep 09 17:34:31.129484 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:31.129455 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:31.129573 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:31.129521 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:31.129573 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:31.129548 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:31.251480 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:31.251012 2580 patch_prober.go:28] interesting pod/e2e-autores-37448c-openshift-default-696795cd84-xngk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 17:34:31.251480 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:31.251075 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 17:34:32.069593 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:32.069558 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" event={"ID":"2e21e9a6-f10d-4cae-984c-4c681b6b0452","Type":"ContainerStarted","Data":"5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9"} Sep 09 17:34:32.069835 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:32.069645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" event={"ID":"2e21e9a6-f10d-4cae-984c-4c681b6b0452","Type":"ContainerStarted","Data":"0d53ef33222c9f078627d104e75d58267d29ff8b029fd9a36e6794ebd2e745d0"} Sep 09 17:34:32.112176 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:32.112126 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" podStartSLOduration=2.112109933 podStartE2EDuration="2.112109933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:32.109777987 +0000 UTC m=+1066.458139461" watchObservedRunningTime="2026-09-09 17:34:32.112109933 +0000 UTC m=+1066.460471405" Sep 09 17:34:32.251496 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:32.251431 2580 patch_prober.go:28] interesting pod/e2e-autores-37448c-openshift-default-696795cd84-xngk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 17:34:32.252030 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:32.251496 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 17:34:32.824445 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:32.824405 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:32.835245 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:32.835206 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:33.075023 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:33.074940 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:33.075953 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:33.075932 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:33.251513 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:33.251461 2580 patch_prober.go:28] interesting pod/e2e-autores-37448c-openshift-default-696795cd84-xngk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 17:34:33.252017 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:33.251528 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 17:34:34.251487 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:34.251054 2580 patch_prober.go:28] interesting pod/e2e-autores-37448c-openshift-default-696795cd84-xngk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 17:34:34.251487 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:34.251119 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 17:34:35.251956 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:35.251911 2580 patch_prober.go:28] interesting pod/e2e-autores-37448c-openshift-default-696795cd84-xngk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Sep 09 17:34:35.252442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:35.251977 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Sep 09 17:34:36.050349 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.050312 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55"] Sep 09 17:34:36.057397 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.057369 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:36.060048 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.060011 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-53da12-dockercfg-bxwsm\"" Sep 09 17:34:36.060048 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.060035 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-53da12\"" Sep 09 17:34:36.064586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.064550 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55"] Sep 09 17:34:36.218662 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.218629 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba70543-9d3a-482d-9871-eb00cbeb81de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55\" (UID: \"7ba70543-9d3a-482d-9871-eb00cbeb81de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:36.218892 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.218700 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7tq\" (UniqueName: \"kubernetes.io/projected/7ba70543-9d3a-482d-9871-eb00cbeb81de-kube-api-access-qp7tq\") pod \"payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55\" (UID: \"7ba70543-9d3a-482d-9871-eb00cbeb81de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:36.255580 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.255554 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:36.256009 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.255987 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:36.256649 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.256632 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:34:36.320076 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.319985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba70543-9d3a-482d-9871-eb00cbeb81de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55\" (UID: \"7ba70543-9d3a-482d-9871-eb00cbeb81de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:36.320076 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.320038 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7tq\" (UniqueName: \"kubernetes.io/projected/7ba70543-9d3a-482d-9871-eb00cbeb81de-kube-api-access-qp7tq\") pod \"payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55\" (UID: \"7ba70543-9d3a-482d-9871-eb00cbeb81de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:36.320662 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.320643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba70543-9d3a-482d-9871-eb00cbeb81de-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55\" (UID: \"7ba70543-9d3a-482d-9871-eb00cbeb81de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:36.327821 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.327794 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7tq\" (UniqueName: \"kubernetes.io/projected/7ba70543-9d3a-482d-9871-eb00cbeb81de-kube-api-access-qp7tq\") pod \"payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55\" (UID: \"7ba70543-9d3a-482d-9871-eb00cbeb81de\") " pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:36.369224 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.369194 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:36.643752 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:36.643663 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55"] Sep 09 17:34:36.646702 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:36.646663 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba70543_9d3a_482d_9871_eb00cbeb81de.slice/crio-5d15169cbaa29aaaab34e0824a5678054cb8e6d0c43ec1f09fe33fcf6a68290c WatchSource:0}: Error finding container 5d15169cbaa29aaaab34e0824a5678054cb8e6d0c43ec1f09fe33fcf6a68290c: Status 404 returned error can't find the container with id 5d15169cbaa29aaaab34e0824a5678054cb8e6d0c43ec1f09fe33fcf6a68290c Sep 09 17:34:37.100284 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.100253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" event={"ID":"7ba70543-9d3a-482d-9871-eb00cbeb81de","Type":"ContainerStarted","Data":"d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010"} Sep 09 17:34:37.100480 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.100311 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" event={"ID":"7ba70543-9d3a-482d-9871-eb00cbeb81de","Type":"ContainerStarted","Data":"5d15169cbaa29aaaab34e0824a5678054cb8e6d0c43ec1f09fe33fcf6a68290c"} Sep 09 17:34:37.100480 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.100342 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:37.118481 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.118434 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" podStartSLOduration=1.118420151 podStartE2EDuration="1.118420151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:37.115555644 +0000 UTC m=+1071.463917116" watchObservedRunningTime="2026-09-09 17:34:37.118420151 +0000 UTC m=+1071.466781653" Sep 09 17:34:37.963769 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.963728 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk"] Sep 09 17:34:37.967992 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.967966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" Sep 09 17:34:37.971216 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.971089 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 17:34:37.971341 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.971319 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\"" Sep 09 17:34:37.972977 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:37.972948 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk"] Sep 09 17:34:38.140802 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:38.140766 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8ad0f-632e-478e-9533-3b8b45c711f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-399f043b-w5bwk\" (UID: \"9cd8ad0f-632e-478e-9533-3b8b45c711f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" Sep 09 17:34:38.241584 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:38.241506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8ad0f-632e-478e-9533-3b8b45c711f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-399f043b-w5bwk\" (UID: \"9cd8ad0f-632e-478e-9533-3b8b45c711f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" Sep 09 17:34:38.242220 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:38.242198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8ad0f-632e-478e-9533-3b8b45c711f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-399f043b-w5bwk\" (UID: \"9cd8ad0f-632e-478e-9533-3b8b45c711f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" Sep 09 17:34:38.283617 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:38.283591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" Sep 09 17:34:38.560167 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:38.560134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk"] Sep 09 17:34:38.564652 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:38.564542 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd8ad0f_632e_478e_9533_3b8b45c711f8.slice/crio-b4201b48b1330c09c0508420929acb75c98e56477b4214473ceafcad38a39f99 WatchSource:0}: Error finding container b4201b48b1330c09c0508420929acb75c98e56477b4214473ceafcad38a39f99: Status 404 returned error can't find the container with id b4201b48b1330c09c0508420929acb75c98e56477b4214473ceafcad38a39f99 Sep 09 17:34:39.120560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:39.120522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" event={"ID":"9cd8ad0f-632e-478e-9533-3b8b45c711f8","Type":"ContainerStarted","Data":"b4201b48b1330c09c0508420929acb75c98e56477b4214473ceafcad38a39f99"} Sep 09 17:34:39.885992 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:39.885956 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:34:43.163607 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:43.163567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" event={"ID":"9cd8ad0f-632e-478e-9533-3b8b45c711f8","Type":"ContainerStarted","Data":"ea4cfcd38e62c6844964549b9965ce753cdd4c94f5f0b9b611ad52d2a98f6d95"} Sep 09 17:34:43.178864 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:43.178815 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" podStartSLOduration=2.3535133950000002 podStartE2EDuration="6.178799079s" totalImagesPullingTime="3.825285684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:34:43.176955501 +0000 UTC m=+1077.525317054" watchObservedRunningTime="2026-09-09 17:34:43.178799079 +0000 UTC m=+1077.527160552" Sep 09 17:34:44.171206 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:44.171172 2580 generic.go:356] "Generic (PLEG): container finished" podID="9cd8ad0f-632e-478e-9533-3b8b45c711f8" containerID="ea4cfcd38e62c6844964549b9965ce753cdd4c94f5f0b9b611ad52d2a98f6d95" exitCode=0 Sep 09 17:34:44.171610 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:44.171253 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" event={"ID":"9cd8ad0f-632e-478e-9533-3b8b45c711f8","Type":"ContainerDied","Data":"ea4cfcd38e62c6844964549b9965ce753cdd4c94f5f0b9b611ad52d2a98f6d95"} Sep 09 17:34:45.360741 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:45.360682 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" Sep 09 17:34:45.515396 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:45.515362 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8ad0f-632e-478e-9533-3b8b45c711f8-maas-api-service-ca\") pod \"9cd8ad0f-632e-478e-9533-3b8b45c711f8\" (UID: \"9cd8ad0f-632e-478e-9533-3b8b45c711f8\") " Sep 09 17:34:45.515682 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:45.515660 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd8ad0f-632e-478e-9533-3b8b45c711f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9cd8ad0f-632e-478e-9533-3b8b45c711f8" (UID: "9cd8ad0f-632e-478e-9533-3b8b45c711f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:34:45.616581 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:45.616550 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd8ad0f-632e-478e-9533-3b8b45c711f8-maas-api-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:34:46.184416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:46.184385 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" Sep 09 17:34:46.184416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:46.184407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk" event={"ID":"9cd8ad0f-632e-478e-9533-3b8b45c711f8","Type":"ContainerDied","Data":"b4201b48b1330c09c0508420929acb75c98e56477b4214473ceafcad38a39f99"} Sep 09 17:34:46.184599 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:46.184455 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4201b48b1330c09c0508420929acb75c98e56477b4214473ceafcad38a39f99" Sep 09 17:34:48.114534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:48.114499 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:34:49.115873 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:49.115816 2580 patch_prober.go:28] interesting pod/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 09 17:34:49.115873 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:49.115873 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 09 17:34:51.217120 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:51.217089 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerID="5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9" exitCode=0 Sep 09 17:34:51.217534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:51.217165 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" event={"ID":"2e21e9a6-f10d-4cae-984c-4c681b6b0452","Type":"ContainerDied","Data":"5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9"} Sep 09 17:34:51.217583 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:51.217572 2580 scope.go:117] "RemoveContainer" containerID="5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9" Sep 09 17:34:51.218671 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:51.218640 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:51.218817 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:51.218758 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:51.218817 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:51.218810 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:34:52.224345 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:52.224310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" event={"ID":"2e21e9a6-f10d-4cae-984c-4c681b6b0452","Type":"ContainerStarted","Data":"da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1"} Sep 09 17:34:52.824786 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:52.824744 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:52.825020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:52.824986 2580 patch_prober.go:28] interesting pod/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 17:34:52.825147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:52.825047 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 17:34:53.824683 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:53.824645 2580 patch_prober.go:28] interesting pod/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 17:34:53.825152 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:53.824727 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 17:34:54.403721 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.403676 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft"] Sep 09 17:34:54.403971 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.403939 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" podUID="4811475c-3148-40f3-92dc-41223183ada6" containerName="payload-processing" containerID="cri-o://96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56" gracePeriod=30 Sep 09 17:34:54.552942 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.552876 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk"] Sep 09 17:34:54.556848 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.556793 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-399f043b-w5bwk"] Sep 09 17:34:54.749945 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.749630 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:54.812877 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.812841 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4811475c-3148-40f3-92dc-41223183ada6-plugins-config-volume\") pod \"4811475c-3148-40f3-92dc-41223183ada6\" (UID: \"4811475c-3148-40f3-92dc-41223183ada6\") " Sep 09 17:34:54.813040 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.812907 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st5pf\" (UniqueName: \"kubernetes.io/projected/4811475c-3148-40f3-92dc-41223183ada6-kube-api-access-st5pf\") pod \"4811475c-3148-40f3-92dc-41223183ada6\" (UID: \"4811475c-3148-40f3-92dc-41223183ada6\") " Sep 09 17:34:54.813556 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.813530 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4811475c-3148-40f3-92dc-41223183ada6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4811475c-3148-40f3-92dc-41223183ada6" (UID: "4811475c-3148-40f3-92dc-41223183ada6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:34:54.815210 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.815188 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4811475c-3148-40f3-92dc-41223183ada6-kube-api-access-st5pf" (OuterVolumeSpecName: "kube-api-access-st5pf") pod "4811475c-3148-40f3-92dc-41223183ada6" (UID: "4811475c-3148-40f3-92dc-41223183ada6"). InnerVolumeSpecName "kube-api-access-st5pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:34:54.825199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.825170 2580 patch_prober.go:28] interesting pod/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 17:34:54.825552 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.825223 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 17:34:54.914161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.914125 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4811475c-3148-40f3-92dc-41223183ada6-plugins-config-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:34:54.914161 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:54.914156 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st5pf\" (UniqueName: \"kubernetes.io/projected/4811475c-3148-40f3-92dc-41223183ada6-kube-api-access-st5pf\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:34:55.253740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.250085 2580 generic.go:356] "Generic (PLEG): container finished" podID="4811475c-3148-40f3-92dc-41223183ada6" containerID="96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56" exitCode=0 Sep 09 17:34:55.253740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.250219 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" event={"ID":"4811475c-3148-40f3-92dc-41223183ada6","Type":"ContainerDied","Data":"96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56"} Sep 09 17:34:55.253740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.250268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" event={"ID":"4811475c-3148-40f3-92dc-41223183ada6","Type":"ContainerDied","Data":"0be2e0b16d8a6d3cffcd437b28df6d92923e223ec9a0361a523e46196323e3e4"} Sep 09 17:34:55.253740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.250299 2580 scope.go:117] "RemoveContainer" containerID="96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56" Sep 09 17:34:55.253740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.250468 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft" Sep 09 17:34:55.270637 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.270557 2580 scope.go:117] "RemoveContainer" containerID="96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56" Sep 09 17:34:55.270938 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:34:55.270911 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56\": container with ID starting with 96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56 not found: ID does not exist" containerID="96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56" Sep 09 17:34:55.271083 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.270946 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56"} err="failed to get container status \"96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56\": rpc error: code = NotFound desc = could not find container \"96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56\": container with ID starting with 96b63facb240a246a6ffc8e7f88b790b1dac26720b4a6e56939c4ea711926b56 not found: ID does not exist" Sep 09 17:34:55.281518 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.281497 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft"] Sep 09 17:34:55.286956 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.286937 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-399f043b-fbf77c7bc-9m5ft"] Sep 09 17:34:55.824561 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.824526 2580 patch_prober.go:28] interesting pod/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 17:34:55.824735 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:55.824587 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 17:34:56.137907 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:56.137817 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4811475c-3148-40f3-92dc-41223183ada6" path="/var/lib/kubelet/pods/4811475c-3148-40f3-92dc-41223183ada6/volumes" Sep 09 17:34:56.138340 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:56.138311 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd8ad0f-632e-478e-9533-3b8b45c711f8" path="/var/lib/kubelet/pods/9cd8ad0f-632e-478e-9533-3b8b45c711f8/volumes" Sep 09 17:34:56.824440 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:56.824401 2580 patch_prober.go:28] interesting pod/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 09 17:34:56.824618 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:56.824457 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 09 17:34:57.827294 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:57.827256 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:57.827764 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:57.827581 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:57.828212 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:57.828194 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:34:58.149599 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.149530 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7"] Sep 09 17:34:58.150021 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.150007 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4811475c-3148-40f3-92dc-41223183ada6" containerName="payload-processing" Sep 09 17:34:58.150071 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.150024 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4811475c-3148-40f3-92dc-41223183ada6" containerName="payload-processing" Sep 09 17:34:58.150071 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.150053 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd8ad0f-632e-478e-9533-3b8b45c711f8" containerName="revoke-keys" Sep 09 17:34:58.150071 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.150059 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd8ad0f-632e-478e-9533-3b8b45c711f8" containerName="revoke-keys" Sep 09 17:34:58.150162 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.150123 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4811475c-3148-40f3-92dc-41223183ada6" containerName="payload-processing" Sep 09 17:34:58.150162 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.150133 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd8ad0f-632e-478e-9533-3b8b45c711f8" containerName="revoke-keys" Sep 09 17:34:58.154528 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.154509 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.157748 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.157722 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-1277f005-serving-cert\"" Sep 09 17:34:58.157936 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.157915 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 17:34:58.158064 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.157745 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 17:34:58.158064 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.157841 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bxbtq\"" Sep 09 17:34:58.158064 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.157831 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 17:34:58.164047 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.164018 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7"] Sep 09 17:34:58.249367 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.249336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-metrics-certs\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.249494 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.249382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.249494 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.249402 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9jj\" (UniqueName: \"kubernetes.io/projected/525d3704-e6a0-48bf-a91c-41c2751b5f75-kube-api-access-hj9jj\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.249580 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.249554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/525d3704-e6a0-48bf-a91c-41c2751b5f75-openshift-service-ca\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.350828 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.350792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/525d3704-e6a0-48bf-a91c-41c2751b5f75-openshift-service-ca\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.350994 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.350859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-metrics-certs\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.350994 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.350896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.350994 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.350918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9jj\" (UniqueName: \"kubernetes.io/projected/525d3704-e6a0-48bf-a91c-41c2751b5f75-kube-api-access-hj9jj\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.351192 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:34:58.351100 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-serving-cert: secret "maas-api-e2e-migrate-1277f005-serving-cert" not found Sep 09 17:34:58.351192 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:34:58.351150 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls podName:525d3704-e6a0-48bf-a91c-41c2751b5f75 nodeName:}" failed. No retries permitted until 2026-09-09 17:34:58.851134144 +0000 UTC m=+1093.199495607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls") pod "maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" (UID: "525d3704-e6a0-48bf-a91c-41c2751b5f75") : secret "maas-api-e2e-migrate-1277f005-serving-cert" not found Sep 09 17:34:58.351479 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.351456 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/525d3704-e6a0-48bf-a91c-41c2751b5f75-openshift-service-ca\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.353299 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.353278 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-metrics-certs\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.359490 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.359464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9jj\" (UniqueName: \"kubernetes.io/projected/525d3704-e6a0-48bf-a91c-41c2751b5f75-kube-api-access-hj9jj\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.855342 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.855308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:58.857574 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:58.857552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls\") pod \"maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:59.068187 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:59.068152 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:34:59.339240 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:59.339205 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7"] Sep 09 17:34:59.341551 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:34:59.341524 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525d3704_e6a0_48bf_a91c_41c2751b5f75.slice/crio-c5440f34c9b601c0cfb47791c36ab4995ee8396cf9201289e6b4cf0c6805a97e WatchSource:0}: Error finding container c5440f34c9b601c0cfb47791c36ab4995ee8396cf9201289e6b4cf0c6805a97e: Status 404 returned error can't find the container with id c5440f34c9b601c0cfb47791c36ab4995ee8396cf9201289e6b4cf0c6805a97e Sep 09 17:34:59.785133 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:34:59.785100 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:35:00.287485 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:00.287453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" event={"ID":"525d3704-e6a0-48bf-a91c-41c2751b5f75","Type":"ContainerStarted","Data":"c5440f34c9b601c0cfb47791c36ab4995ee8396cf9201289e6b4cf0c6805a97e"} Sep 09 17:35:01.294613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:01.294575 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" event={"ID":"525d3704-e6a0-48bf-a91c-41c2751b5f75","Type":"ContainerStarted","Data":"09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c"} Sep 09 17:35:01.295019 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:01.294753 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:35:01.310855 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:01.310810 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" podStartSLOduration=1.6850735289999998 podStartE2EDuration="3.310798285s" totalImagesPullingTime="1.625724756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:01.309705832 +0000 UTC m=+1095.658067307" watchObservedRunningTime="2026-09-09 17:35:01.310798285 +0000 UTC m=+1095.659159758" Sep 09 17:35:07.305489 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:07.305461 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:35:12.684844 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:12.684811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:35:27.494650 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:27.494619 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:35:29.480822 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.480789 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7"] Sep 09 17:35:29.481575 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.481543 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" podUID="525d3704-e6a0-48bf-a91c-41c2751b5f75" containerName="maas-api" containerID="cri-o://09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c" gracePeriod=30 Sep 09 17:35:29.564420 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:35:29.564383 2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525d3704_e6a0_48bf_a91c_41c2751b5f75.slice/crio-09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c.scope\": RecentStats: unable to find data in memory cache]" Sep 09 17:35:29.758608 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.758576 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:35:29.855426 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.855395 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-metrics-certs\") pod \"525d3704-e6a0-48bf-a91c-41c2751b5f75\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " Sep 09 17:35:29.855613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.855488 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/525d3704-e6a0-48bf-a91c-41c2751b5f75-openshift-service-ca\") pod \"525d3704-e6a0-48bf-a91c-41c2751b5f75\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " Sep 09 17:35:29.855613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.855517 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj9jj\" (UniqueName: \"kubernetes.io/projected/525d3704-e6a0-48bf-a91c-41c2751b5f75-kube-api-access-hj9jj\") pod \"525d3704-e6a0-48bf-a91c-41c2751b5f75\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " Sep 09 17:35:29.855613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.855535 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls\") pod \"525d3704-e6a0-48bf-a91c-41c2751b5f75\" (UID: \"525d3704-e6a0-48bf-a91c-41c2751b5f75\") " Sep 09 17:35:29.855890 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.855863 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525d3704-e6a0-48bf-a91c-41c2751b5f75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "525d3704-e6a0-48bf-a91c-41c2751b5f75" (UID: "525d3704-e6a0-48bf-a91c-41c2751b5f75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:35:29.857871 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.857844 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "525d3704-e6a0-48bf-a91c-41c2751b5f75" (UID: "525d3704-e6a0-48bf-a91c-41c2751b5f75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:35:29.857871 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.857860 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "525d3704-e6a0-48bf-a91c-41c2751b5f75" (UID: "525d3704-e6a0-48bf-a91c-41c2751b5f75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:35:29.858008 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.857951 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525d3704-e6a0-48bf-a91c-41c2751b5f75-kube-api-access-hj9jj" (OuterVolumeSpecName: "kube-api-access-hj9jj") pod "525d3704-e6a0-48bf-a91c-41c2751b5f75" (UID: "525d3704-e6a0-48bf-a91c-41c2751b5f75"). InnerVolumeSpecName "kube-api-access-hj9jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:35:29.956786 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.956746 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:35:29.956786 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.956775 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/525d3704-e6a0-48bf-a91c-41c2751b5f75-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:35:29.956786 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.956785 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj9jj\" (UniqueName: \"kubernetes.io/projected/525d3704-e6a0-48bf-a91c-41c2751b5f75-kube-api-access-hj9jj\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:35:29.956786 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:29.956793 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/525d3704-e6a0-48bf-a91c-41c2751b5f75-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:35:30.477978 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.477872 2580 generic.go:356] "Generic (PLEG): container finished" podID="525d3704-e6a0-48bf-a91c-41c2751b5f75" containerID="09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c" exitCode=0 Sep 09 17:35:30.477978 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.477920 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" Sep 09 17:35:30.477978 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.477953 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" event={"ID":"525d3704-e6a0-48bf-a91c-41c2751b5f75","Type":"ContainerDied","Data":"09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c"} Sep 09 17:35:30.478226 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.478016 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7" event={"ID":"525d3704-e6a0-48bf-a91c-41c2751b5f75","Type":"ContainerDied","Data":"c5440f34c9b601c0cfb47791c36ab4995ee8396cf9201289e6b4cf0c6805a97e"} Sep 09 17:35:30.478226 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.478046 2580 scope.go:117] "RemoveContainer" containerID="09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c" Sep 09 17:35:30.488460 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.488233 2580 scope.go:117] "RemoveContainer" containerID="09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c" Sep 09 17:35:30.488675 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:35:30.488524 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c\": container with ID starting with 09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c not found: ID does not exist" containerID="09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c" Sep 09 17:35:30.488675 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.488549 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c"} err="failed to get container status \"09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c\": rpc error: code = NotFound desc = could not find container \"09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c\": container with ID starting with 09e4b35be75e693dc31302ad9045754d24c167f3f8e84c14cda146c45c2ffe9c not found: ID does not exist" Sep 09 17:35:30.497660 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.497637 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7"] Sep 09 17:35:30.504272 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:30.504247 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1277f005-59c8b95488-zb4j7"] Sep 09 17:35:32.132836 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:32.132798 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="525d3704-e6a0-48bf-a91c-41c2751b5f75" path="/var/lib/kubelet/pods/525d3704-e6a0-48bf-a91c-41c2751b5f75/volumes" Sep 09 17:35:33.994498 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:33.994464 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m"] Sep 09 17:35:33.995088 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:33.995070 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="525d3704-e6a0-48bf-a91c-41c2751b5f75" containerName="maas-api" Sep 09 17:35:33.995088 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:33.995088 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="525d3704-e6a0-48bf-a91c-41c2751b5f75" containerName="maas-api" Sep 09 17:35:33.995171 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:33.995158 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="525d3704-e6a0-48bf-a91c-41c2751b5f75" containerName="maas-api" Sep 09 17:35:34.001565 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.001536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.006185 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.005402 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-50082ca1-serving-cert\"" Sep 09 17:35:34.006185 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.005486 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 17:35:34.006185 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.005545 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bxbtq\"" Sep 09 17:35:34.006185 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.005493 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 17:35:34.006185 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.005869 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 17:35:34.010191 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.010158 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m"] Sep 09 17:35:34.092192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.092156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.092333 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.092226 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6r8\" (UniqueName: \"kubernetes.io/projected/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-kube-api-access-hx6r8\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.092333 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.092250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-openshift-service-ca\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.092333 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.092268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-metrics-certs\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.193818 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.193781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.194028 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.193858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6r8\" (UniqueName: \"kubernetes.io/projected/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-kube-api-access-hx6r8\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.194028 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.193878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-openshift-service-ca\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.194028 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.193896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-metrics-certs\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.194170 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:35:34.194029 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-serving-cert: secret "maas-api-e2e-ait-50082ca1-serving-cert" not found Sep 09 17:35:34.194170 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:35:34.194104 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls podName:3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c nodeName:}" failed. No retries permitted until 2026-09-09 17:35:34.694083173 +0000 UTC m=+1129.042444638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls") pod "maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" (UID: "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c") : secret "maas-api-e2e-ait-50082ca1-serving-cert" not found Sep 09 17:35:34.194498 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.194475 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-openshift-service-ca\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.196294 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.196273 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-metrics-certs\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.202354 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.202329 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6r8\" (UniqueName: \"kubernetes.io/projected/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-kube-api-access-hx6r8\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.690159 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.690121 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2"] Sep 09 17:35:34.694837 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.694819 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:34.697435 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.697415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-50082ca1-dockercfg-2zzvm\"" Sep 09 17:35:34.697739 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.697693 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-50082ca1\"" Sep 09 17:35:34.698003 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.697984 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.700536 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.700514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls\") pod \"maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:34.703867 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.703846 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2"] Sep 09 17:35:34.798703 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.798666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb0499c8-a97f-4304-85de-fef7a6910ffb-plugins-config-volume\") pod \"payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2\" (UID: \"fb0499c8-a97f-4304-85de-fef7a6910ffb\") " pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:34.798902 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.798729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8rzg\" (UniqueName: \"kubernetes.io/projected/fb0499c8-a97f-4304-85de-fef7a6910ffb-kube-api-access-g8rzg\") pod \"payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2\" (UID: \"fb0499c8-a97f-4304-85de-fef7a6910ffb\") " pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:34.900145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.900110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb0499c8-a97f-4304-85de-fef7a6910ffb-plugins-config-volume\") pod \"payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2\" (UID: \"fb0499c8-a97f-4304-85de-fef7a6910ffb\") " pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:34.900145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.900155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8rzg\" (UniqueName: \"kubernetes.io/projected/fb0499c8-a97f-4304-85de-fef7a6910ffb-kube-api-access-g8rzg\") pod \"payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2\" (UID: \"fb0499c8-a97f-4304-85de-fef7a6910ffb\") " pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:34.900820 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.900797 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb0499c8-a97f-4304-85de-fef7a6910ffb-plugins-config-volume\") pod \"payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2\" (UID: \"fb0499c8-a97f-4304-85de-fef7a6910ffb\") " pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:34.908083 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.908054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8rzg\" (UniqueName: \"kubernetes.io/projected/fb0499c8-a97f-4304-85de-fef7a6910ffb-kube-api-access-g8rzg\") pod \"payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2\" (UID: \"fb0499c8-a97f-4304-85de-fef7a6910ffb\") " pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:34.914814 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:34.914791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:35.016882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:35.016850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:35.190975 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:35.190948 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m"] Sep 09 17:35:35.192372 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:35:35.192346 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc1ed5d_dc41_4212_98e1_e1c2a5c28c5c.slice/crio-e5e30d507753f9bb7c88f6215a139a797153d543e56d67fcdf0a66f2aadb00d4 WatchSource:0}: Error finding container e5e30d507753f9bb7c88f6215a139a797153d543e56d67fcdf0a66f2aadb00d4: Status 404 returned error can't find the container with id e5e30d507753f9bb7c88f6215a139a797153d543e56d67fcdf0a66f2aadb00d4 Sep 09 17:35:35.288995 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:35.288966 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2"] Sep 09 17:35:35.291070 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:35:35.291038 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0499c8_a97f_4304_85de_fef7a6910ffb.slice/crio-e0951ec11742915f9bb26d30a1a8655d674a1b963e8e24828d5e239f061f69af WatchSource:0}: Error finding container e0951ec11742915f9bb26d30a1a8655d674a1b963e8e24828d5e239f061f69af: Status 404 returned error can't find the container with id e0951ec11742915f9bb26d30a1a8655d674a1b963e8e24828d5e239f061f69af Sep 09 17:35:35.510002 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:35.509962 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" event={"ID":"fb0499c8-a97f-4304-85de-fef7a6910ffb","Type":"ContainerStarted","Data":"fa146a45b62c5103c9b95dd5a935732a473fa0688cc0bc478fcfa89c3bcc707a"} Sep 09 17:35:35.510212 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:35.510030 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" event={"ID":"fb0499c8-a97f-4304-85de-fef7a6910ffb","Type":"ContainerStarted","Data":"e0951ec11742915f9bb26d30a1a8655d674a1b963e8e24828d5e239f061f69af"} Sep 09 17:35:35.510212 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:35.510059 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:35.511286 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:35.511264 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" event={"ID":"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c","Type":"ContainerStarted","Data":"e5e30d507753f9bb7c88f6215a139a797153d543e56d67fcdf0a66f2aadb00d4"} Sep 09 17:35:35.527315 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:35.527262 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" podStartSLOduration=1.527244927 podStartE2EDuration="1.527244927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:35.524144809 +0000 UTC m=+1129.872506279" watchObservedRunningTime="2026-09-09 17:35:35.527244927 +0000 UTC m=+1129.875606405" Sep 09 17:35:36.518554 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:36.518515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" event={"ID":"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c","Type":"ContainerStarted","Data":"2824cf45f5d503ba85d7da605a1909c647a9493404c8cb236d924a9fdebeb9ff"} Sep 09 17:35:36.519031 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:36.518763 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:36.535354 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:36.535307 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" podStartSLOduration=3.098771985 podStartE2EDuration="3.535292987s" totalImagesPullingTime="436.521002ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:36.533628536 +0000 UTC m=+1130.881990007" watchObservedRunningTime="2026-09-09 17:35:36.535292987 +0000 UTC m=+1130.883654461" Sep 09 17:35:42.532189 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:42.532157 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:35:46.098291 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:46.098255 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:35:46.521385 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:46.521354 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:35:58.557763 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:58.557730 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw"] Sep 09 17:35:58.563315 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:58.563293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" Sep 09 17:35:58.565873 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:58.565850 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9x8dp\"" Sep 09 17:35:58.567660 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:58.567631 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw"] Sep 09 17:35:58.741433 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:58.741399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc42cdd-ab85-4518-9fc3-dfdafdc0caca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw\" (UID: \"afc42cdd-ab85-4518-9fc3-dfdafdc0caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" Sep 09 17:35:58.842939 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:58.842844 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc42cdd-ab85-4518-9fc3-dfdafdc0caca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw\" (UID: \"afc42cdd-ab85-4518-9fc3-dfdafdc0caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" Sep 09 17:35:58.843659 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:58.843632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc42cdd-ab85-4518-9fc3-dfdafdc0caca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw\" (UID: \"afc42cdd-ab85-4518-9fc3-dfdafdc0caca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" Sep 09 17:35:58.875663 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:58.875632 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" Sep 09 17:35:59.149047 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:59.149021 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw"] Sep 09 17:35:59.151227 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:35:59.151199 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafc42cdd_ab85_4518_9fc3_dfdafdc0caca.slice/crio-3ad51410f82e7c3d45618791f15d3cb07101c49753b10cf03e8fba3c6361248c WatchSource:0}: Error finding container 3ad51410f82e7c3d45618791f15d3cb07101c49753b10cf03e8fba3c6361248c: Status 404 returned error can't find the container with id 3ad51410f82e7c3d45618791f15d3cb07101c49753b10cf03e8fba3c6361248c Sep 09 17:35:59.661321 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:59.661287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" event={"ID":"afc42cdd-ab85-4518-9fc3-dfdafdc0caca","Type":"ContainerStarted","Data":"b04224dee7c1086cbc1c77026ca7cd9e64b19ce47503bc934d9b4dce4ca1819d"} Sep 09 17:35:59.661736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:59.661343 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" event={"ID":"afc42cdd-ab85-4518-9fc3-dfdafdc0caca","Type":"ContainerStarted","Data":"3ad51410f82e7c3d45618791f15d3cb07101c49753b10cf03e8fba3c6361248c"} Sep 09 17:35:59.679645 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:35:59.679591 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" podStartSLOduration=1.6795791279999999 podStartE2EDuration="1.679579128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:35:59.677757556 +0000 UTC m=+1154.026119030" watchObservedRunningTime="2026-09-09 17:35:59.679579128 +0000 UTC m=+1154.027940600" Sep 09 17:36:00.668874 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:00.668837 2580 generic.go:356] "Generic (PLEG): container finished" podID="afc42cdd-ab85-4518-9fc3-dfdafdc0caca" containerID="b04224dee7c1086cbc1c77026ca7cd9e64b19ce47503bc934d9b4dce4ca1819d" exitCode=0 Sep 09 17:36:00.669260 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:00.668877 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" event={"ID":"afc42cdd-ab85-4518-9fc3-dfdafdc0caca","Type":"ContainerDied","Data":"b04224dee7c1086cbc1c77026ca7cd9e64b19ce47503bc934d9b4dce4ca1819d"} Sep 09 17:36:01.811094 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:01.811061 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" Sep 09 17:36:01.973893 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:01.973800 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc42cdd-ab85-4518-9fc3-dfdafdc0caca-maas-api-service-ca\") pod \"afc42cdd-ab85-4518-9fc3-dfdafdc0caca\" (UID: \"afc42cdd-ab85-4518-9fc3-dfdafdc0caca\") " Sep 09 17:36:01.974129 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:01.974105 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afc42cdd-ab85-4518-9fc3-dfdafdc0caca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "afc42cdd-ab85-4518-9fc3-dfdafdc0caca" (UID: "afc42cdd-ab85-4518-9fc3-dfdafdc0caca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:36:02.075223 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:02.075181 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afc42cdd-ab85-4518-9fc3-dfdafdc0caca-maas-api-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:02.682644 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:02.682614 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" Sep 09 17:36:02.682859 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:02.682649 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw" event={"ID":"afc42cdd-ab85-4518-9fc3-dfdafdc0caca","Type":"ContainerDied","Data":"3ad51410f82e7c3d45618791f15d3cb07101c49753b10cf03e8fba3c6361248c"} Sep 09 17:36:02.682859 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:02.682695 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ad51410f82e7c3d45618791f15d3cb07101c49753b10cf03e8fba3c6361248c" Sep 09 17:36:03.035929 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:03.035892 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:36:04.476756 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.476699 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m"] Sep 09 17:36:04.477252 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.476952 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" podUID="3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" containerName="maas-api" containerID="cri-o://2824cf45f5d503ba85d7da605a1909c647a9493404c8cb236d924a9fdebeb9ff" gracePeriod=30 Sep 09 17:36:04.531949 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.531898 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2"] Sep 09 17:36:04.532475 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.532431 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" podUID="fb0499c8-a97f-4304-85de-fef7a6910ffb" containerName="payload-processing" containerID="cri-o://fa146a45b62c5103c9b95dd5a935732a473fa0688cc0bc478fcfa89c3bcc707a" gracePeriod=30 Sep 09 17:36:04.685584 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.685546 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw"] Sep 09 17:36:04.687671 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.687639 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-50082ca1-6lnvw"] Sep 09 17:36:04.697262 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.696791 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb0499c8-a97f-4304-85de-fef7a6910ffb" containerID="fa146a45b62c5103c9b95dd5a935732a473fa0688cc0bc478fcfa89c3bcc707a" exitCode=0 Sep 09 17:36:04.697262 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.696957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" event={"ID":"fb0499c8-a97f-4304-85de-fef7a6910ffb","Type":"ContainerDied","Data":"fa146a45b62c5103c9b95dd5a935732a473fa0688cc0bc478fcfa89c3bcc707a"} Sep 09 17:36:04.699137 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.699050 2580 generic.go:356] "Generic (PLEG): container finished" podID="3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" containerID="2824cf45f5d503ba85d7da605a1909c647a9493404c8cb236d924a9fdebeb9ff" exitCode=0 Sep 09 17:36:04.699137 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.699129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" event={"ID":"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c","Type":"ContainerDied","Data":"2824cf45f5d503ba85d7da605a1909c647a9493404c8cb236d924a9fdebeb9ff"} Sep 09 17:36:04.800779 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.800745 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:36:04.801154 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.801129 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx6r8\" (UniqueName: \"kubernetes.io/projected/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-kube-api-access-hx6r8\") pod \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " Sep 09 17:36:04.801273 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.801184 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-metrics-certs\") pod \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " Sep 09 17:36:04.801273 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.801253 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls\") pod \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " Sep 09 17:36:04.801389 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.801290 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-openshift-service-ca\") pod \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\" (UID: \"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c\") " Sep 09 17:36:04.801770 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.801727 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" (UID: "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:36:04.803416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.803391 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-kube-api-access-hx6r8" (OuterVolumeSpecName: "kube-api-access-hx6r8") pod "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" (UID: "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c"). InnerVolumeSpecName "kube-api-access-hx6r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:36:04.803416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.803403 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" (UID: "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:36:04.803608 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.803562 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" (UID: "3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:36:04.824919 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.824892 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:36:04.902657 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.902626 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb0499c8-a97f-4304-85de-fef7a6910ffb-plugins-config-volume\") pod \"fb0499c8-a97f-4304-85de-fef7a6910ffb\" (UID: \"fb0499c8-a97f-4304-85de-fef7a6910ffb\") " Sep 09 17:36:04.902836 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.902674 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8rzg\" (UniqueName: \"kubernetes.io/projected/fb0499c8-a97f-4304-85de-fef7a6910ffb-kube-api-access-g8rzg\") pod \"fb0499c8-a97f-4304-85de-fef7a6910ffb\" (UID: \"fb0499c8-a97f-4304-85de-fef7a6910ffb\") " Sep 09 17:36:04.902999 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.902986 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:04.903043 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.903001 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:04.903043 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.903011 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx6r8\" (UniqueName: \"kubernetes.io/projected/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-kube-api-access-hx6r8\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:04.903043 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.903019 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:04.903043 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.903016 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb0499c8-a97f-4304-85de-fef7a6910ffb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb0499c8-a97f-4304-85de-fef7a6910ffb" (UID: "fb0499c8-a97f-4304-85de-fef7a6910ffb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:36:04.904659 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:04.904638 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb0499c8-a97f-4304-85de-fef7a6910ffb-kube-api-access-g8rzg" (OuterVolumeSpecName: "kube-api-access-g8rzg") pod "fb0499c8-a97f-4304-85de-fef7a6910ffb" (UID: "fb0499c8-a97f-4304-85de-fef7a6910ffb"). InnerVolumeSpecName "kube-api-access-g8rzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:36:05.004513 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.004477 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb0499c8-a97f-4304-85de-fef7a6910ffb-plugins-config-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:05.004513 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.004507 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8rzg\" (UniqueName: \"kubernetes.io/projected/fb0499c8-a97f-4304-85de-fef7a6910ffb-kube-api-access-g8rzg\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:05.706779 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.706743 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" event={"ID":"3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c","Type":"ContainerDied","Data":"e5e30d507753f9bb7c88f6215a139a797153d543e56d67fcdf0a66f2aadb00d4"} Sep 09 17:36:05.707202 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.706756 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m" Sep 09 17:36:05.707202 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.706810 2580 scope.go:117] "RemoveContainer" containerID="2824cf45f5d503ba85d7da605a1909c647a9493404c8cb236d924a9fdebeb9ff" Sep 09 17:36:05.708233 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.708207 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" Sep 09 17:36:05.708330 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.708212 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2" event={"ID":"fb0499c8-a97f-4304-85de-fef7a6910ffb","Type":"ContainerDied","Data":"e0951ec11742915f9bb26d30a1a8655d674a1b963e8e24828d5e239f061f69af"} Sep 09 17:36:05.718776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.718756 2580 scope.go:117] "RemoveContainer" containerID="fa146a45b62c5103c9b95dd5a935732a473fa0688cc0bc478fcfa89c3bcc707a" Sep 09 17:36:05.728586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.728565 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m"] Sep 09 17:36:05.736166 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.736128 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-50082ca1-55bbbd66c5-n4b9m"] Sep 09 17:36:05.745409 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.745336 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2"] Sep 09 17:36:05.748311 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:05.748288 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-50082ca1-789c876cc5-kn6n2"] Sep 09 17:36:06.131304 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:06.131269 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" path="/var/lib/kubelet/pods/3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c/volumes" Sep 09 17:36:06.131639 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:06.131624 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afc42cdd-ab85-4518-9fc3-dfdafdc0caca" path="/var/lib/kubelet/pods/afc42cdd-ab85-4518-9fc3-dfdafdc0caca/volumes" Sep 09 17:36:06.132021 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:06.132000 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb0499c8-a97f-4304-85de-fef7a6910ffb" path="/var/lib/kubelet/pods/fb0499c8-a97f-4304-85de-fef7a6910ffb/volumes" Sep 09 17:36:10.155232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.155136 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf"] Sep 09 17:36:10.155896 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.155876 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afc42cdd-ab85-4518-9fc3-dfdafdc0caca" containerName="revoke-keys" Sep 09 17:36:10.155972 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.155900 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="afc42cdd-ab85-4518-9fc3-dfdafdc0caca" containerName="revoke-keys" Sep 09 17:36:10.155972 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.155930 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb0499c8-a97f-4304-85de-fef7a6910ffb" containerName="payload-processing" Sep 09 17:36:10.155972 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.155937 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb0499c8-a97f-4304-85de-fef7a6910ffb" containerName="payload-processing" Sep 09 17:36:10.155972 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.155960 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" containerName="maas-api" Sep 09 17:36:10.155972 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.155968 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" containerName="maas-api" Sep 09 17:36:10.156177 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.156081 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="afc42cdd-ab85-4518-9fc3-dfdafdc0caca" containerName="revoke-keys" Sep 09 17:36:10.156177 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.156096 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb0499c8-a97f-4304-85de-fef7a6910ffb" containerName="payload-processing" Sep 09 17:36:10.156177 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.156113 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc1ed5d-dc41-4212-98e1-e1c2a5c28c5c" containerName="maas-api" Sep 09 17:36:10.161040 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.161020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.163595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.163570 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0aa81a54-serving-cert\"" Sep 09 17:36:10.163722 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.163685 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bxbtq\"" Sep 09 17:36:10.163991 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.163973 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 17:36:10.165336 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.164999 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 17:36:10.165336 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.165190 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 17:36:10.168724 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.168690 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf"] Sep 09 17:36:10.255654 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.255618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72afa740-d240-4055-9e42-8260bc96f1d1-openshift-service-ca\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.255825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.255733 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-metrics-certs\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.255825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.255785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/72afa740-d240-4055-9e42-8260bc96f1d1-kube-api-access-n5k5l\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.255939 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.255923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-maas-api-tls\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.357110 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.357071 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72afa740-d240-4055-9e42-8260bc96f1d1-openshift-service-ca\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.357300 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.357139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-metrics-certs\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.357300 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.357168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/72afa740-d240-4055-9e42-8260bc96f1d1-kube-api-access-n5k5l\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.357300 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.357290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-maas-api-tls\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.357826 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.357798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72afa740-d240-4055-9e42-8260bc96f1d1-openshift-service-ca\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.359779 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.359755 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-maas-api-tls\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.359864 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.359759 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-metrics-certs\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.364873 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.364848 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/72afa740-d240-4055-9e42-8260bc96f1d1-kube-api-access-n5k5l\") pod \"maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.473987 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.473910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:10.746036 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:10.746011 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf"] Sep 09 17:36:10.747909 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:36:10.747883 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72afa740_d240_4055_9e42_8260bc96f1d1.slice/crio-f99b8f29cced185d341a2982a8da8333263891ee0c6c0ff82507e57e103bbe75 WatchSource:0}: Error finding container f99b8f29cced185d341a2982a8da8333263891ee0c6c0ff82507e57e103bbe75: Status 404 returned error can't find the container with id f99b8f29cced185d341a2982a8da8333263891ee0c6c0ff82507e57e103bbe75 Sep 09 17:36:11.748401 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:11.748368 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" event={"ID":"72afa740-d240-4055-9e42-8260bc96f1d1","Type":"ContainerStarted","Data":"f99b8f29cced185d341a2982a8da8333263891ee0c6c0ff82507e57e103bbe75"} Sep 09 17:36:13.761759 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:13.761695 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" event={"ID":"72afa740-d240-4055-9e42-8260bc96f1d1","Type":"ContainerStarted","Data":"6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b"} Sep 09 17:36:13.762205 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:13.761815 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:13.779740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:13.779684 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" podStartSLOduration=1.673576588 podStartE2EDuration="3.779670481s" totalImagesPullingTime="2.106093893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:36:13.776920787 +0000 UTC m=+1168.125282258" watchObservedRunningTime="2026-09-09 17:36:13.779670481 +0000 UTC m=+1168.128031953" Sep 09 17:36:19.391265 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:19.391227 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:36:19.776669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:19.776639 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:40.641950 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.641911 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf"] Sep 09 17:36:40.642556 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.642157 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" podUID="72afa740-d240-4055-9e42-8260bc96f1d1" containerName="maas-api" containerID="cri-o://6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b" gracePeriod=30 Sep 09 17:36:40.897215 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.897149 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:40.937685 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.936971 2580 generic.go:356] "Generic (PLEG): container finished" podID="72afa740-d240-4055-9e42-8260bc96f1d1" containerID="6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b" exitCode=0 Sep 09 17:36:40.937685 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.937109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" event={"ID":"72afa740-d240-4055-9e42-8260bc96f1d1","Type":"ContainerDied","Data":"6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b"} Sep 09 17:36:40.937685 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.937161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" event={"ID":"72afa740-d240-4055-9e42-8260bc96f1d1","Type":"ContainerDied","Data":"f99b8f29cced185d341a2982a8da8333263891ee0c6c0ff82507e57e103bbe75"} Sep 09 17:36:40.937685 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.937191 2580 scope.go:117] "RemoveContainer" containerID="6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b" Sep 09 17:36:40.937685 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.937355 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf" Sep 09 17:36:40.960049 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.958523 2580 scope.go:117] "RemoveContainer" containerID="6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b" Sep 09 17:36:40.961082 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:36:40.961048 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b\": container with ID starting with 6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b not found: ID does not exist" containerID="6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b" Sep 09 17:36:40.963034 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:40.962955 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b"} err="failed to get container status \"6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b\": rpc error: code = NotFound desc = could not find container \"6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b\": container with ID starting with 6a12f7041c03abd92d2bfc941cac6cb64ce4ccb78b8db8ae5a057f41935a9f4b not found: ID does not exist" Sep 09 17:36:41.057535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.057494 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/72afa740-d240-4055-9e42-8260bc96f1d1-kube-api-access-n5k5l\") pod \"72afa740-d240-4055-9e42-8260bc96f1d1\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " Sep 09 17:36:41.057699 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.057552 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72afa740-d240-4055-9e42-8260bc96f1d1-openshift-service-ca\") pod \"72afa740-d240-4055-9e42-8260bc96f1d1\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " Sep 09 17:36:41.057699 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.057575 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-maas-api-tls\") pod \"72afa740-d240-4055-9e42-8260bc96f1d1\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " Sep 09 17:36:41.057699 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.057655 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-metrics-certs\") pod \"72afa740-d240-4055-9e42-8260bc96f1d1\" (UID: \"72afa740-d240-4055-9e42-8260bc96f1d1\") " Sep 09 17:36:41.057976 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.057945 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72afa740-d240-4055-9e42-8260bc96f1d1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72afa740-d240-4055-9e42-8260bc96f1d1" (UID: "72afa740-d240-4055-9e42-8260bc96f1d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:36:41.059809 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.059780 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "72afa740-d240-4055-9e42-8260bc96f1d1" (UID: "72afa740-d240-4055-9e42-8260bc96f1d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:36:41.059917 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.059862 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72afa740-d240-4055-9e42-8260bc96f1d1" (UID: "72afa740-d240-4055-9e42-8260bc96f1d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:36:41.059966 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.059928 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72afa740-d240-4055-9e42-8260bc96f1d1-kube-api-access-n5k5l" (OuterVolumeSpecName: "kube-api-access-n5k5l") pod "72afa740-d240-4055-9e42-8260bc96f1d1" (UID: "72afa740-d240-4055-9e42-8260bc96f1d1"). InnerVolumeSpecName "kube-api-access-n5k5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:36:41.158704 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.158623 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:41.158704 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.158653 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5k5l\" (UniqueName: \"kubernetes.io/projected/72afa740-d240-4055-9e42-8260bc96f1d1-kube-api-access-n5k5l\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:41.158704 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.158661 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72afa740-d240-4055-9e42-8260bc96f1d1-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:41.158704 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.158670 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72afa740-d240-4055-9e42-8260bc96f1d1-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:36:41.259030 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.258993 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf"] Sep 09 17:36:41.261255 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.261229 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0aa81a54-bfc4f8b49-mgcsf"] Sep 09 17:36:41.997415 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:41.997377 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:36:42.132687 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:42.132643 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72afa740-d240-4055-9e42-8260bc96f1d1" path="/var/lib/kubelet/pods/72afa740-d240-4055-9e42-8260bc96f1d1/volumes" Sep 09 17:36:46.251223 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:46.251195 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:36:46.253389 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:46.253365 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:36:46.254386 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:46.254366 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:36:46.258074 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:46.258054 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:36:56.387056 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:56.387018 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7"] Sep 09 17:36:56.387543 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:56.387225 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" containerID="cri-o://41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915" gracePeriod=30 Sep 09 17:36:57.784944 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:36:57.784904 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:37:01.142952 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.142912 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7"] Sep 09 17:37:01.143564 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.143542 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72afa740-d240-4055-9e42-8260bc96f1d1" containerName="maas-api" Sep 09 17:37:01.143683 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.143566 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="72afa740-d240-4055-9e42-8260bc96f1d1" containerName="maas-api" Sep 09 17:37:01.143769 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.143704 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="72afa740-d240-4055-9e42-8260bc96f1d1" containerName="maas-api" Sep 09 17:37:01.148547 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.148518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.152367 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.152344 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 17:37:01.152511 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.152491 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bxbtq\"" Sep 09 17:37:01.152587 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.152445 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-95cd01-serving-cert\"" Sep 09 17:37:01.152646 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.152493 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 17:37:01.152646 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.152375 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 17:37:01.158937 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.158913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7"] Sep 09 17:37:01.257719 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.257674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-metrics-certs\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.257898 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.257876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-maas-api-tls\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.257943 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.257918 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mmbc\" (UniqueName: \"kubernetes.io/projected/4b1c83e2-98f2-40a3-b76a-156c8cce3639-kube-api-access-9mmbc\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.257979 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.257946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1c83e2-98f2-40a3-b76a-156c8cce3639-openshift-service-ca\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.358472 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.358433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-maas-api-tls\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.358472 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.358473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mmbc\" (UniqueName: \"kubernetes.io/projected/4b1c83e2-98f2-40a3-b76a-156c8cce3639-kube-api-access-9mmbc\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.358739 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.358493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1c83e2-98f2-40a3-b76a-156c8cce3639-openshift-service-ca\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.358739 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.358517 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-metrics-certs\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.359239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.359216 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1c83e2-98f2-40a3-b76a-156c8cce3639-openshift-service-ca\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.361063 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.361038 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-metrics-certs\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.361166 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.361088 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-maas-api-tls\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.366364 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.366337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mmbc\" (UniqueName: \"kubernetes.io/projected/4b1c83e2-98f2-40a3-b76a-156c8cce3639-kube-api-access-9mmbc\") pod \"maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.461906 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.461869 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:01.657442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.657413 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:37:01.742693 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.742666 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7"] Sep 09 17:37:01.744601 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:37:01.744575 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1c83e2_98f2_40a3_b76a_156c8cce3639.slice/crio-72d1c55866ec92bf871fafdcd26429727d11c3727c75cf677b1d80fd5454f265 WatchSource:0}: Error finding container 72d1c55866ec92bf871fafdcd26429727d11c3727c75cf677b1d80fd5454f265: Status 404 returned error can't find the container with id 72d1c55866ec92bf871fafdcd26429727d11c3727c75cf677b1d80fd5454f265 Sep 09 17:37:01.762096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.761975 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istiod-ca-cert\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.762096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762029 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-socket\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.762096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762087 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-podinfo\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.762435 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762121 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-envoy\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.762435 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762156 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-948dq\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-kube-api-access-948dq\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.762435 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762180 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-token\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.762435 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762198 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-data\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.762435 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762271 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-credential-socket\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.762435 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762297 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-certs\") pod \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\" (UID: \"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7\") " Sep 09 17:37:01.763010 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.762927 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:37:01.763304 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.763104 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:37:01.763376 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.763339 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:37:01.763484 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.763460 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:37:01.763580 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.763559 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:37:01.764847 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.764825 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 17:37:01.765061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.765040 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:37:01.765174 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.765104 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-kube-api-access-948dq" (OuterVolumeSpecName: "kube-api-access-948dq") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "kube-api-access-948dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:37:01.765174 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.765139 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" (UID: "25328c87-cc8f-40c7-ab6b-1ab8640fd0a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:37:01.790028 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.789987 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb"] Sep 09 17:37:01.790537 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.790519 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" Sep 09 17:37:01.790624 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.790540 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" Sep 09 17:37:01.790694 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.790670 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" Sep 09 17:37:01.790694 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.790685 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" Sep 09 17:37:01.795113 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.795092 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:01.797807 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.797778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-95cd01\"" Sep 09 17:37:01.797911 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.797877 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-95cd01-dockercfg-798ll\"" Sep 09 17:37:01.804939 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.804913 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb"] Sep 09 17:37:01.864192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbtbr\" (UniqueName: \"kubernetes.io/projected/e56f0469-61af-483b-a533-76c1ee29443d-kube-api-access-gbtbr\") pod \"payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb\" (UID: \"e56f0469-61af-483b-a533-76c1ee29443d\") " pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:01.864403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56f0469-61af-483b-a533-76c1ee29443d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb\" (UID: \"e56f0469-61af-483b-a533-76c1ee29443d\") " pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:01.864403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864358 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.864403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864372 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.864403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864384 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.864403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864396 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.864613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864408 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-948dq\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-kube-api-access-948dq\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.864613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864420 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.864613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864432 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.864613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864445 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.864613 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.864457 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:37:01.965669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.965565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56f0469-61af-483b-a533-76c1ee29443d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb\" (UID: \"e56f0469-61af-483b-a533-76c1ee29443d\") " pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:01.965884 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.965679 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbtbr\" (UniqueName: \"kubernetes.io/projected/e56f0469-61af-483b-a533-76c1ee29443d-kube-api-access-gbtbr\") pod \"payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb\" (UID: \"e56f0469-61af-483b-a533-76c1ee29443d\") " pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:01.966311 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.966289 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56f0469-61af-483b-a533-76c1ee29443d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb\" (UID: \"e56f0469-61af-483b-a533-76c1ee29443d\") " pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:01.973578 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:01.973550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbtbr\" (UniqueName: \"kubernetes.io/projected/e56f0469-61af-483b-a533-76c1ee29443d-kube-api-access-gbtbr\") pod \"payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb\" (UID: \"e56f0469-61af-483b-a533-76c1ee29443d\") " pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:02.078672 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.078627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" event={"ID":"4b1c83e2-98f2-40a3-b76a-156c8cce3639","Type":"ContainerStarted","Data":"72d1c55866ec92bf871fafdcd26429727d11c3727c75cf677b1d80fd5454f265"} Sep 09 17:37:02.080425 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.080398 2580 generic.go:356] "Generic (PLEG): container finished" podID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerID="41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915" exitCode=0 Sep 09 17:37:02.080534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.080434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" event={"ID":"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7","Type":"ContainerDied","Data":"41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915"} Sep 09 17:37:02.080534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.080463 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" Sep 09 17:37:02.080534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.080490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7" event={"ID":"25328c87-cc8f-40c7-ab6b-1ab8640fd0a7","Type":"ContainerDied","Data":"bb664888c870dd35e2ec8350c3253ca1716ab0737acfa6e29fad2c1d3468e562"} Sep 09 17:37:02.080534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.080513 2580 scope.go:117] "RemoveContainer" containerID="41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915" Sep 09 17:37:02.090374 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.090353 2580 scope.go:117] "RemoveContainer" containerID="720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d" Sep 09 17:37:02.099282 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.099244 2580 scope.go:117] "RemoveContainer" containerID="41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915" Sep 09 17:37:02.099733 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:37:02.099687 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915\": container with ID starting with 41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915 not found: ID does not exist" containerID="41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915" Sep 09 17:37:02.099847 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.099745 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915"} err="failed to get container status \"41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915\": rpc error: code = NotFound desc = could not find container \"41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915\": container with ID starting with 41ce5f3835ca7b0fd3f0ce1e27d7c968b22dd90f259fb4d2311cb3af46550915 not found: ID does not exist" Sep 09 17:37:02.099847 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.099773 2580 scope.go:117] "RemoveContainer" containerID="720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d" Sep 09 17:37:02.100210 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:37:02.100185 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d\": container with ID starting with 720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d not found: ID does not exist" containerID="720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d" Sep 09 17:37:02.100287 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.100217 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d"} err="failed to get container status \"720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d\": rpc error: code = NotFound desc = could not find container \"720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d\": container with ID starting with 720960a44e8fcbb339b10af285f64fd21a7c3cb607d80e6fea0842d6db31624d not found: ID does not exist" Sep 09 17:37:02.102364 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.102347 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7"] Sep 09 17:37:02.106389 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.106362 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-37448c-openshift-default-696795cd84-xngk7"] Sep 09 17:37:02.106491 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.106474 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:02.132654 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.132615 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" path="/var/lib/kubelet/pods/25328c87-cc8f-40c7-ab6b-1ab8640fd0a7/volumes" Sep 09 17:37:02.381109 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:02.381066 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb"] Sep 09 17:37:02.382524 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:37:02.382499 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56f0469_61af_483b_a533_76c1ee29443d.slice/crio-d715a6e0e64ba8f82c83b1e1e144abaca4239fcf530a58d6e3aac775813d1995 WatchSource:0}: Error finding container d715a6e0e64ba8f82c83b1e1e144abaca4239fcf530a58d6e3aac775813d1995: Status 404 returned error can't find the container with id d715a6e0e64ba8f82c83b1e1e144abaca4239fcf530a58d6e3aac775813d1995 Sep 09 17:37:03.087901 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:03.087864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" event={"ID":"e56f0469-61af-483b-a533-76c1ee29443d","Type":"ContainerStarted","Data":"43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4"} Sep 09 17:37:03.088115 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:03.087917 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" event={"ID":"e56f0469-61af-483b-a533-76c1ee29443d","Type":"ContainerStarted","Data":"d715a6e0e64ba8f82c83b1e1e144abaca4239fcf530a58d6e3aac775813d1995"} Sep 09 17:37:03.088115 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:03.087976 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:03.089603 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:03.089582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" event={"ID":"4b1c83e2-98f2-40a3-b76a-156c8cce3639","Type":"ContainerStarted","Data":"b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d"} Sep 09 17:37:03.089689 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:03.089674 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:03.104884 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:03.104835 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" podStartSLOduration=2.104817919 podStartE2EDuration="2.104817919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:37:03.103634573 +0000 UTC m=+1217.451996047" watchObservedRunningTime="2026-09-09 17:37:03.104817919 +0000 UTC m=+1217.453179390" Sep 09 17:37:03.119170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:03.119110 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" podStartSLOduration=1.712599802 podStartE2EDuration="2.119093603s" totalImagesPullingTime="406.493801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:37:03.118459645 +0000 UTC m=+1217.466821118" watchObservedRunningTime="2026-09-09 17:37:03.119093603 +0000 UTC m=+1217.467455118" Sep 09 17:37:09.100817 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:09.100786 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:37:12.790316 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:12.790282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:37:14.098267 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:14.098232 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:37:28.583853 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:28.583820 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:37:41.583552 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:41.583473 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:37:52.792080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:37:52.792046 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:38:09.289356 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:09.289317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:38:20.360361 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.360320 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9"] Sep 09 17:38:20.361087 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.361069 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" Sep 09 17:38:20.361143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.361091 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="25328c87-cc8f-40c7-ab6b-1ab8640fd0a7" containerName="istio-proxy" Sep 09 17:38:20.364832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.364806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.367429 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.367400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-71f58f92-serving-cert\"" Sep 09 17:38:20.372947 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.372923 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9"] Sep 09 17:38:20.419417 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.419386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-metrics-certs\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.419583 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.419429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.419583 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.419532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e755d90-1353-4082-bf5c-22153e6edad6-openshift-service-ca\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.419583 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.419578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5t77\" (UniqueName: \"kubernetes.io/projected/3e755d90-1353-4082-bf5c-22153e6edad6-kube-api-access-j5t77\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.520701 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.520667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-metrics-certs\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.520898 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.520729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.520898 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.520766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e755d90-1353-4082-bf5c-22153e6edad6-openshift-service-ca\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.520898 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.520793 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5t77\" (UniqueName: \"kubernetes.io/projected/3e755d90-1353-4082-bf5c-22153e6edad6-kube-api-access-j5t77\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.520898 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:38:20.520865 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-serving-cert: secret "maas-api-e2e-mt-71f58f92-serving-cert" not found Sep 09 17:38:20.521064 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:38:20.520939 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls podName:3e755d90-1353-4082-bf5c-22153e6edad6 nodeName:}" failed. No retries permitted until 2026-09-09 17:38:21.020920588 +0000 UTC m=+1295.369282050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls") pod "maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" (UID: "3e755d90-1353-4082-bf5c-22153e6edad6") : secret "maas-api-e2e-mt-71f58f92-serving-cert" not found Sep 09 17:38:20.521426 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.521403 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e755d90-1353-4082-bf5c-22153e6edad6-openshift-service-ca\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.523077 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.523058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-metrics-certs\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:20.528909 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:20.528875 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5t77\" (UniqueName: \"kubernetes.io/projected/3e755d90-1353-4082-bf5c-22153e6edad6-kube-api-access-j5t77\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:21.026978 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:21.026942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:21.029385 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:21.029354 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls\") pod \"maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:21.279646 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:21.279543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:21.551759 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:21.551726 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9"] Sep 09 17:38:21.553510 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:38:21.553483 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e755d90_1353_4082_bf5c_22153e6edad6.slice/crio-19d20cbd33436219fe8429f197d823f76b76138680b8572906c16551273fce5c WatchSource:0}: Error finding container 19d20cbd33436219fe8429f197d823f76b76138680b8572906c16551273fce5c: Status 404 returned error can't find the container with id 19d20cbd33436219fe8429f197d823f76b76138680b8572906c16551273fce5c Sep 09 17:38:21.562732 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:21.562681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" event={"ID":"3e755d90-1353-4082-bf5c-22153e6edad6","Type":"ContainerStarted","Data":"19d20cbd33436219fe8429f197d823f76b76138680b8572906c16551273fce5c"} Sep 09 17:38:22.570583 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:22.570488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" event={"ID":"3e755d90-1353-4082-bf5c-22153e6edad6","Type":"ContainerStarted","Data":"65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce"} Sep 09 17:38:22.570963 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:22.570609 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:22.586794 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:22.586742 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" podStartSLOduration=1.8679293590000001 podStartE2EDuration="2.58672218s" totalImagesPullingTime="718.792821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:38:22.586421503 +0000 UTC m=+1296.934782977" watchObservedRunningTime="2026-09-09 17:38:22.58672218 +0000 UTC m=+1296.935083683" Sep 09 17:38:24.663371 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:24.663338 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7"] Sep 09 17:38:24.663760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:24.663587 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" podUID="4b1c83e2-98f2-40a3-b76a-156c8cce3639" containerName="maas-api" containerID="cri-o://b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d" gracePeriod=30 Sep 09 17:38:24.707085 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:24.707055 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:38:24.714212 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:24.714185 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb"] Sep 09 17:38:24.714432 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:24.714405 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" podUID="e56f0469-61af-483b-a533-76c1ee29443d" containerName="payload-processing" containerID="cri-o://43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4" gracePeriod=30 Sep 09 17:38:24.989925 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:24.989893 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:38:24.992343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:24.992325 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:38:25.069578 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.069545 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1c83e2-98f2-40a3-b76a-156c8cce3639-openshift-service-ca\") pod \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " Sep 09 17:38:25.069766 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.069586 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-maas-api-tls\") pod \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " Sep 09 17:38:25.069766 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.069618 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-metrics-certs\") pod \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " Sep 09 17:38:25.069766 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.069701 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mmbc\" (UniqueName: \"kubernetes.io/projected/4b1c83e2-98f2-40a3-b76a-156c8cce3639-kube-api-access-9mmbc\") pod \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\" (UID: \"4b1c83e2-98f2-40a3-b76a-156c8cce3639\") " Sep 09 17:38:25.069766 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.069756 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56f0469-61af-483b-a533-76c1ee29443d-plugins-config-volume\") pod \"e56f0469-61af-483b-a533-76c1ee29443d\" (UID: \"e56f0469-61af-483b-a533-76c1ee29443d\") " Sep 09 17:38:25.069993 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.069793 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbtbr\" (UniqueName: \"kubernetes.io/projected/e56f0469-61af-483b-a533-76c1ee29443d-kube-api-access-gbtbr\") pod \"e56f0469-61af-483b-a533-76c1ee29443d\" (UID: \"e56f0469-61af-483b-a533-76c1ee29443d\") " Sep 09 17:38:25.070072 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.069959 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1c83e2-98f2-40a3-b76a-156c8cce3639-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b1c83e2-98f2-40a3-b76a-156c8cce3639" (UID: "4b1c83e2-98f2-40a3-b76a-156c8cce3639"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:38:25.070145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.070127 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1c83e2-98f2-40a3-b76a-156c8cce3639-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:38:25.070200 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.070161 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56f0469-61af-483b-a533-76c1ee29443d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e56f0469-61af-483b-a533-76c1ee29443d" (UID: "e56f0469-61af-483b-a533-76c1ee29443d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:38:25.071755 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.071699 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b1c83e2-98f2-40a3-b76a-156c8cce3639" (UID: "4b1c83e2-98f2-40a3-b76a-156c8cce3639"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:38:25.071851 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.071781 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1c83e2-98f2-40a3-b76a-156c8cce3639-kube-api-access-9mmbc" (OuterVolumeSpecName: "kube-api-access-9mmbc") pod "4b1c83e2-98f2-40a3-b76a-156c8cce3639" (UID: "4b1c83e2-98f2-40a3-b76a-156c8cce3639"). InnerVolumeSpecName "kube-api-access-9mmbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:38:25.072122 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.072104 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b1c83e2-98f2-40a3-b76a-156c8cce3639" (UID: "4b1c83e2-98f2-40a3-b76a-156c8cce3639"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:38:25.072171 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.072130 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56f0469-61af-483b-a533-76c1ee29443d-kube-api-access-gbtbr" (OuterVolumeSpecName: "kube-api-access-gbtbr") pod "e56f0469-61af-483b-a533-76c1ee29443d" (UID: "e56f0469-61af-483b-a533-76c1ee29443d"). InnerVolumeSpecName "kube-api-access-gbtbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:38:25.171013 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.170975 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mmbc\" (UniqueName: \"kubernetes.io/projected/4b1c83e2-98f2-40a3-b76a-156c8cce3639-kube-api-access-9mmbc\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:38:25.171013 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.171005 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e56f0469-61af-483b-a533-76c1ee29443d-plugins-config-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:38:25.171013 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.171014 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbtbr\" (UniqueName: \"kubernetes.io/projected/e56f0469-61af-483b-a533-76c1ee29443d-kube-api-access-gbtbr\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:38:25.171233 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.171023 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:38:25.171233 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.171033 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b1c83e2-98f2-40a3-b76a-156c8cce3639-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:38:25.589898 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.589863 2580 generic.go:356] "Generic (PLEG): container finished" podID="e56f0469-61af-483b-a533-76c1ee29443d" containerID="43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4" exitCode=0 Sep 09 17:38:25.590102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.589907 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" Sep 09 17:38:25.590102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.589947 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" event={"ID":"e56f0469-61af-483b-a533-76c1ee29443d","Type":"ContainerDied","Data":"43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4"} Sep 09 17:38:25.590102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.590006 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb" event={"ID":"e56f0469-61af-483b-a533-76c1ee29443d","Type":"ContainerDied","Data":"d715a6e0e64ba8f82c83b1e1e144abaca4239fcf530a58d6e3aac775813d1995"} Sep 09 17:38:25.590102 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.590037 2580 scope.go:117] "RemoveContainer" containerID="43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4" Sep 09 17:38:25.591357 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.591332 2580 generic.go:356] "Generic (PLEG): container finished" podID="4b1c83e2-98f2-40a3-b76a-156c8cce3639" containerID="b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d" exitCode=0 Sep 09 17:38:25.591469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.591371 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" Sep 09 17:38:25.591469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.591423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" event={"ID":"4b1c83e2-98f2-40a3-b76a-156c8cce3639","Type":"ContainerDied","Data":"b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d"} Sep 09 17:38:25.591551 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.591473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7" event={"ID":"4b1c83e2-98f2-40a3-b76a-156c8cce3639","Type":"ContainerDied","Data":"72d1c55866ec92bf871fafdcd26429727d11c3727c75cf677b1d80fd5454f265"} Sep 09 17:38:25.600466 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.600451 2580 scope.go:117] "RemoveContainer" containerID="43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4" Sep 09 17:38:25.600791 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:38:25.600765 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4\": container with ID starting with 43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4 not found: ID does not exist" containerID="43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4" Sep 09 17:38:25.600882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.600799 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4"} err="failed to get container status \"43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4\": rpc error: code = NotFound desc = could not find container \"43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4\": container with ID starting with 43ea9a2b3e34cf6e1ac2e071e4b22fb95a857ee8dc8632c972a82492873cc8e4 not found: ID does not exist" Sep 09 17:38:25.600882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.600817 2580 scope.go:117] "RemoveContainer" containerID="b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d" Sep 09 17:38:25.609554 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.609536 2580 scope.go:117] "RemoveContainer" containerID="b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d" Sep 09 17:38:25.609869 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:38:25.609848 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d\": container with ID starting with b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d not found: ID does not exist" containerID="b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d" Sep 09 17:38:25.609931 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.609874 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d"} err="failed to get container status \"b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d\": rpc error: code = NotFound desc = could not find container \"b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d\": container with ID starting with b399fcadb1ae3d426f7587b035535fbad401b5fc2617be0014adb7d7f193f41d not found: ID does not exist" Sep 09 17:38:25.618060 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.617998 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb"] Sep 09 17:38:25.621238 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.621213 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-95cd01-764b98477c-bb9kb"] Sep 09 17:38:25.632810 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.632752 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7"] Sep 09 17:38:25.636316 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:25.636289 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-95cd01-7468d4cbc5-zwfq7"] Sep 09 17:38:26.132693 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:26.132650 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b1c83e2-98f2-40a3-b76a-156c8cce3639" path="/var/lib/kubelet/pods/4b1c83e2-98f2-40a3-b76a-156c8cce3639/volumes" Sep 09 17:38:26.133219 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:26.133197 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e56f0469-61af-483b-a533-76c1ee29443d" path="/var/lib/kubelet/pods/e56f0469-61af-483b-a533-76c1ee29443d/volumes" Sep 09 17:38:28.581827 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:28.581792 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:38:32.169796 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.169751 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp"] Sep 09 17:38:32.170469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.170451 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e56f0469-61af-483b-a533-76c1ee29443d" containerName="payload-processing" Sep 09 17:38:32.170532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.170471 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e56f0469-61af-483b-a533-76c1ee29443d" containerName="payload-processing" Sep 09 17:38:32.170532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.170490 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b1c83e2-98f2-40a3-b76a-156c8cce3639" containerName="maas-api" Sep 09 17:38:32.170532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.170496 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b1c83e2-98f2-40a3-b76a-156c8cce3639" containerName="maas-api" Sep 09 17:38:32.170649 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.170577 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e56f0469-61af-483b-a533-76c1ee29443d" containerName="payload-processing" Sep 09 17:38:32.170649 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.170590 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b1c83e2-98f2-40a3-b76a-156c8cce3639" containerName="maas-api" Sep 09 17:38:32.175773 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.175705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.178624 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.178604 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-12bac2-openshift-default-dockercfg-j2xln\"" Sep 09 17:38:32.185705 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.185684 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp"] Sep 09 17:38:32.350386 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350343 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-certs\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.350386 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545e1736-efb2-4c41-92fb-63195bd37784-istiod-ca-cert\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.350690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-data\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.350690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350467 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-credential-socket\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.350690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545e1736-efb2-4c41-92fb-63195bd37784-istio-podinfo\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.350690 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-socket\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.350862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-envoy\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.350862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-istio-token\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.350862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.350810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th6zm\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-kube-api-access-th6zm\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.451729 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th6zm\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-kube-api-access-th6zm\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.451729 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-certs\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.451729 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545e1736-efb2-4c41-92fb-63195bd37784-istiod-ca-cert\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-data\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-credential-socket\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451816 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545e1736-efb2-4c41-92fb-63195bd37784-istio-podinfo\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-socket\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-envoy\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.451978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-istio-token\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452341 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.452103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-certs\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452341 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.452177 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-data\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452341 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.452312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-socket\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452503 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.452343 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-credential-socket\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.452761 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.452737 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545e1736-efb2-4c41-92fb-63195bd37784-istiod-ca-cert\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.454347 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.454323 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-envoy\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.454486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.454463 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545e1736-efb2-4c41-92fb-63195bd37784-istio-podinfo\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.459428 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.459408 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-istio-token\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.459781 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.459760 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th6zm\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-kube-api-access-th6zm\") pod \"e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.487126 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.487098 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:32.765065 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.765034 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp"] Sep 09 17:38:32.766893 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:38:32.766860 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545e1736_efb2_4c41_92fb_63195bd37784.slice/crio-0284e1acede7b87a0492bb77d659ac815693ded906586154c18328acb8cb279e WatchSource:0}: Error finding container 0284e1acede7b87a0492bb77d659ac815693ded906586154c18328acb8cb279e: Status 404 returned error can't find the container with id 0284e1acede7b87a0492bb77d659ac815693ded906586154c18328acb8cb279e Sep 09 17:38:32.769474 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.769445 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:38:32.769548 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.769513 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:38:32.769548 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:32.769542 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 09 17:38:33.647967 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:33.647930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" event={"ID":"545e1736-efb2-4c41-92fb-63195bd37784","Type":"ContainerStarted","Data":"96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff"} Sep 09 17:38:33.648340 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:33.647987 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" event={"ID":"545e1736-efb2-4c41-92fb-63195bd37784","Type":"ContainerStarted","Data":"0284e1acede7b87a0492bb77d659ac815693ded906586154c18328acb8cb279e"} Sep 09 17:38:33.668207 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:33.668158 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" podStartSLOduration=1.6681443009999999 podStartE2EDuration="1.668144301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:38:33.66717238 +0000 UTC m=+1308.015533853" watchObservedRunningTime="2026-09-09 17:38:33.668144301 +0000 UTC m=+1308.016505773" Sep 09 17:38:34.487724 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:34.487687 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:34.493050 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:34.493028 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:34.654082 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:34.654049 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:34.655057 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:34.655037 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:38:40.779895 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:40.779862 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:38:42.506442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.506405 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d"] Sep 09 17:38:42.506905 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.506635 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" podUID="9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3" containerName="payload-pre-processing" containerID="cri-o://720e87f60d1ba6b8bb81b0db254dd14b6fe6279857c924d644937c21c6e38241" gracePeriod=30 Sep 09 17:38:42.704896 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.704864 2580 generic.go:356] "Generic (PLEG): container finished" podID="9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3" containerID="720e87f60d1ba6b8bb81b0db254dd14b6fe6279857c924d644937c21c6e38241" exitCode=0 Sep 09 17:38:42.705069 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.704945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" event={"ID":"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3","Type":"ContainerDied","Data":"720e87f60d1ba6b8bb81b0db254dd14b6fe6279857c924d644937c21c6e38241"} Sep 09 17:38:42.778154 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.778124 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:38:42.849756 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.849699 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-plugins-config-volume\") pod \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\" (UID: \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\") " Sep 09 17:38:42.849933 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.849838 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67k8m\" (UniqueName: \"kubernetes.io/projected/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-kube-api-access-67k8m\") pod \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\" (UID: \"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3\") " Sep 09 17:38:42.850120 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.850085 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3" (UID: "9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:38:42.851834 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.851800 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-kube-api-access-67k8m" (OuterVolumeSpecName: "kube-api-access-67k8m") pod "9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3" (UID: "9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3"). InnerVolumeSpecName "kube-api-access-67k8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:38:42.951222 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.951197 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-plugins-config-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:38:42.951222 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:42.951220 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67k8m\" (UniqueName: \"kubernetes.io/projected/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3-kube-api-access-67k8m\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:38:43.711755 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:43.711706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" event={"ID":"9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3","Type":"ContainerDied","Data":"1b1710692eafd466ee945a50b2cf0e5bd5db0925ab91ef9351ea23fed6f908a0"} Sep 09 17:38:43.712211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:43.711785 2580 scope.go:117] "RemoveContainer" containerID="720e87f60d1ba6b8bb81b0db254dd14b6fe6279857c924d644937c21c6e38241" Sep 09 17:38:43.712211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:43.711730 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d" Sep 09 17:38:43.736460 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:43.736434 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d"] Sep 09 17:38:43.740538 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:43.740511 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ea996c-78655bf88b-tc87d"] Sep 09 17:38:44.131643 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:44.131611 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3" path="/var/lib/kubelet/pods/9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3/volumes" Sep 09 17:38:49.890304 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:38:49.890268 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:39:00.123238 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.123201 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55"] Sep 09 17:39:00.123823 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.123412 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" podUID="7ba70543-9d3a-482d-9871-eb00cbeb81de" containerName="payload-processing" containerID="cri-o://d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010" gracePeriod=30 Sep 09 17:39:00.393313 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.393284 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:39:00.507752 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.507707 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp7tq\" (UniqueName: \"kubernetes.io/projected/7ba70543-9d3a-482d-9871-eb00cbeb81de-kube-api-access-qp7tq\") pod \"7ba70543-9d3a-482d-9871-eb00cbeb81de\" (UID: \"7ba70543-9d3a-482d-9871-eb00cbeb81de\") " Sep 09 17:39:00.507928 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.507763 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba70543-9d3a-482d-9871-eb00cbeb81de-plugins-config-volume\") pod \"7ba70543-9d3a-482d-9871-eb00cbeb81de\" (UID: \"7ba70543-9d3a-482d-9871-eb00cbeb81de\") " Sep 09 17:39:00.508135 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.508103 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba70543-9d3a-482d-9871-eb00cbeb81de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ba70543-9d3a-482d-9871-eb00cbeb81de" (UID: "7ba70543-9d3a-482d-9871-eb00cbeb81de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:39:00.509772 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.509752 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba70543-9d3a-482d-9871-eb00cbeb81de-kube-api-access-qp7tq" (OuterVolumeSpecName: "kube-api-access-qp7tq") pod "7ba70543-9d3a-482d-9871-eb00cbeb81de" (UID: "7ba70543-9d3a-482d-9871-eb00cbeb81de"). InnerVolumeSpecName "kube-api-access-qp7tq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:39:00.608673 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.608635 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7tq\" (UniqueName: \"kubernetes.io/projected/7ba70543-9d3a-482d-9871-eb00cbeb81de-kube-api-access-qp7tq\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:00.608673 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.608667 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba70543-9d3a-482d-9871-eb00cbeb81de-plugins-config-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:00.813521 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.813487 2580 generic.go:356] "Generic (PLEG): container finished" podID="7ba70543-9d3a-482d-9871-eb00cbeb81de" containerID="d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010" exitCode=0 Sep 09 17:39:00.813818 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.813538 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" Sep 09 17:39:00.813818 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.813577 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" event={"ID":"7ba70543-9d3a-482d-9871-eb00cbeb81de","Type":"ContainerDied","Data":"d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010"} Sep 09 17:39:00.813818 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.813634 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55" event={"ID":"7ba70543-9d3a-482d-9871-eb00cbeb81de","Type":"ContainerDied","Data":"5d15169cbaa29aaaab34e0824a5678054cb8e6d0c43ec1f09fe33fcf6a68290c"} Sep 09 17:39:00.813818 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.813662 2580 scope.go:117] "RemoveContainer" containerID="d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010" Sep 09 17:39:00.822803 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.822784 2580 scope.go:117] "RemoveContainer" containerID="d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010" Sep 09 17:39:00.823109 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:00.823090 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010\": container with ID starting with d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010 not found: ID does not exist" containerID="d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010" Sep 09 17:39:00.823172 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.823117 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010"} err="failed to get container status \"d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010\": rpc error: code = NotFound desc = could not find container \"d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010\": container with ID starting with d7df1746f3c588b50201b40fe5da3884da4edaaaf80b2c0d5728b9ae1f026010 not found: ID does not exist" Sep 09 17:39:00.836452 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.836421 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55"] Sep 09 17:39:00.840290 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:00.840265 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-53da12-85dfc69f47-hft55"] Sep 09 17:39:02.131593 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.131550 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba70543-9d3a-482d-9871-eb00cbeb81de" path="/var/lib/kubelet/pods/7ba70543-9d3a-482d-9871-eb00cbeb81de/volumes" Sep 09 17:39:02.201422 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.201389 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9"] Sep 09 17:39:02.201694 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.201645 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" podUID="3e755d90-1353-4082-bf5c-22153e6edad6" containerName="maas-api" containerID="cri-o://65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce" gracePeriod=30 Sep 09 17:39:02.457080 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.457051 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:39:02.527361 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.527327 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls\") pod \"3e755d90-1353-4082-bf5c-22153e6edad6\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " Sep 09 17:39:02.527557 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.527376 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5t77\" (UniqueName: \"kubernetes.io/projected/3e755d90-1353-4082-bf5c-22153e6edad6-kube-api-access-j5t77\") pod \"3e755d90-1353-4082-bf5c-22153e6edad6\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " Sep 09 17:39:02.527557 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.527414 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e755d90-1353-4082-bf5c-22153e6edad6-openshift-service-ca\") pod \"3e755d90-1353-4082-bf5c-22153e6edad6\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " Sep 09 17:39:02.527557 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.527443 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-metrics-certs\") pod \"3e755d90-1353-4082-bf5c-22153e6edad6\" (UID: \"3e755d90-1353-4082-bf5c-22153e6edad6\") " Sep 09 17:39:02.527855 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.527828 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e755d90-1353-4082-bf5c-22153e6edad6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3e755d90-1353-4082-bf5c-22153e6edad6" (UID: "3e755d90-1353-4082-bf5c-22153e6edad6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:39:02.529486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.529462 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3e755d90-1353-4082-bf5c-22153e6edad6" (UID: "3e755d90-1353-4082-bf5c-22153e6edad6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:39:02.529591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.529479 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e755d90-1353-4082-bf5c-22153e6edad6-kube-api-access-j5t77" (OuterVolumeSpecName: "kube-api-access-j5t77") pod "3e755d90-1353-4082-bf5c-22153e6edad6" (UID: "3e755d90-1353-4082-bf5c-22153e6edad6"). InnerVolumeSpecName "kube-api-access-j5t77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:39:02.529591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.529498 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3e755d90-1353-4082-bf5c-22153e6edad6" (UID: "3e755d90-1353-4082-bf5c-22153e6edad6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:39:02.628566 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.628533 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:02.628566 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.628560 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5t77\" (UniqueName: \"kubernetes.io/projected/3e755d90-1353-4082-bf5c-22153e6edad6-kube-api-access-j5t77\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:02.628566 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.628569 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e755d90-1353-4082-bf5c-22153e6edad6-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:02.628823 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.628578 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e755d90-1353-4082-bf5c-22153e6edad6-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:02.827608 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.827574 2580 generic.go:356] "Generic (PLEG): container finished" podID="3e755d90-1353-4082-bf5c-22153e6edad6" containerID="65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce" exitCode=0 Sep 09 17:39:02.827819 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.827641 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" Sep 09 17:39:02.827819 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.827654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" event={"ID":"3e755d90-1353-4082-bf5c-22153e6edad6","Type":"ContainerDied","Data":"65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce"} Sep 09 17:39:02.827819 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.827701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9" event={"ID":"3e755d90-1353-4082-bf5c-22153e6edad6","Type":"ContainerDied","Data":"19d20cbd33436219fe8429f197d823f76b76138680b8572906c16551273fce5c"} Sep 09 17:39:02.827819 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.827738 2580 scope.go:117] "RemoveContainer" containerID="65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce" Sep 09 17:39:02.838214 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.838194 2580 scope.go:117] "RemoveContainer" containerID="65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce" Sep 09 17:39:02.838518 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:02.838498 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce\": container with ID starting with 65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce not found: ID does not exist" containerID="65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce" Sep 09 17:39:02.838593 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.838525 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce"} err="failed to get container status \"65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce\": rpc error: code = NotFound desc = could not find container \"65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce\": container with ID starting with 65e1be783aa795c54935981b002089a1bb6c7cff93ec210c6997c37cf17d9fce not found: ID does not exist" Sep 09 17:39:02.850609 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.850579 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9"] Sep 09 17:39:02.856213 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:02.856190 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-71f58f92-7894f7798d-hgtw9"] Sep 09 17:39:04.130920 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:04.130880 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e755d90-1353-4082-bf5c-22153e6edad6" path="/var/lib/kubelet/pods/3e755d90-1353-4082-bf5c-22153e6edad6/volumes" Sep 09 17:39:05.791106 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:05.791067 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:39:06.183410 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:06.183325 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp"] Sep 09 17:39:06.183542 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:06.183526 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" podUID="545e1736-efb2-4c41-92fb-63195bd37784" containerName="istio-proxy" containerID="cri-o://96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff" gracePeriod=30 Sep 09 17:39:06.998603 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:06.998572 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn"] Sep 09 17:39:06.999061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:06.998791 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" containerID="cri-o://da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1" gracePeriod=30 Sep 09 17:39:11.437931 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.437905 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:39:11.520355 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520331 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th6zm\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-kube-api-access-th6zm\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520488 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520374 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-certs\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520488 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520406 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545e1736-efb2-4c41-92fb-63195bd37784-istio-podinfo\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520607 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520509 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545e1736-efb2-4c41-92fb-63195bd37784-istiod-ca-cert\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520607 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520575 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-credential-socket\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520607 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520600 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-envoy\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520785 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520628 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-istio-token\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520785 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520674 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-data\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520785 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520696 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-socket\") pod \"545e1736-efb2-4c41-92fb-63195bd37784\" (UID: \"545e1736-efb2-4c41-92fb-63195bd37784\") " Sep 09 17:39:11.520785 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520750 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:11.520989 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520924 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:11.520989 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520956 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-data" (OuterVolumeSpecName: "istio-data") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:11.520989 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.520975 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545e1736-efb2-4c41-92fb-63195bd37784-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:39:11.521137 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.521125 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.521192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.521153 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.521192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.521167 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/545e1736-efb2-4c41-92fb-63195bd37784-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.521192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.521179 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.521192 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.521121 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:11.522651 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.522621 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-kube-api-access-th6zm" (OuterVolumeSpecName: "kube-api-access-th6zm") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "kube-api-access-th6zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:39:11.522791 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.522702 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:11.522969 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.522947 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/545e1736-efb2-4c41-92fb-63195bd37784-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 17:39:11.523051 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.523005 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-istio-token" (OuterVolumeSpecName: "istio-token") pod "545e1736-efb2-4c41-92fb-63195bd37784" (UID: "545e1736-efb2-4c41-92fb-63195bd37784"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:39:11.622317 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.622282 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th6zm\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-kube-api-access-th6zm\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.622317 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.622314 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/545e1736-efb2-4c41-92fb-63195bd37784-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.622317 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.622323 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.622317 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.622331 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/545e1736-efb2-4c41-92fb-63195bd37784-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.622535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.622339 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/545e1736-efb2-4c41-92fb-63195bd37784-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:11.736572 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.736542 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6"] Sep 09 17:39:11.737061 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737043 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3" containerName="payload-pre-processing" Sep 09 17:39:11.737158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737064 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3" containerName="payload-pre-processing" Sep 09 17:39:11.737158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737078 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545e1736-efb2-4c41-92fb-63195bd37784" containerName="istio-proxy" Sep 09 17:39:11.737158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737086 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="545e1736-efb2-4c41-92fb-63195bd37784" containerName="istio-proxy" Sep 09 17:39:11.737158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737106 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba70543-9d3a-482d-9871-eb00cbeb81de" containerName="payload-processing" Sep 09 17:39:11.737158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737114 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba70543-9d3a-482d-9871-eb00cbeb81de" containerName="payload-processing" Sep 09 17:39:11.737158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737143 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e755d90-1353-4082-bf5c-22153e6edad6" containerName="maas-api" Sep 09 17:39:11.737158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737150 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e755d90-1353-4082-bf5c-22153e6edad6" containerName="maas-api" Sep 09 17:39:11.737486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737289 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="545e1736-efb2-4c41-92fb-63195bd37784" containerName="istio-proxy" Sep 09 17:39:11.737486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737303 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba70543-9d3a-482d-9871-eb00cbeb81de" containerName="payload-processing" Sep 09 17:39:11.737486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737312 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e755d90-1353-4082-bf5c-22153e6edad6" containerName="maas-api" Sep 09 17:39:11.737486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.737323 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f44b3cc-b013-4b2b-b0fd-0aa69b7ad9f3" containerName="payload-pre-processing" Sep 09 17:39:11.740793 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.740772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.743433 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.743399 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 17:39:11.743433 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.743425 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bxbtq\"" Sep 09 17:39:11.743611 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.743399 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 17:39:11.743611 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.743487 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-480517-serving-cert\"" Sep 09 17:39:11.743870 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.743851 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 17:39:11.749860 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.749836 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6"] Sep 09 17:39:11.824114 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.824033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx45k\" (UniqueName: \"kubernetes.io/projected/f150c868-970d-4322-b70b-8115200d69fb-kube-api-access-mx45k\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.824114 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.824095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f150c868-970d-4322-b70b-8115200d69fb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.824307 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.824126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.824307 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.824151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.883478 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.883440 2580 generic.go:356] "Generic (PLEG): container finished" podID="545e1736-efb2-4c41-92fb-63195bd37784" containerID="96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff" exitCode=0 Sep 09 17:39:11.883643 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.883488 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" Sep 09 17:39:11.883643 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.883526 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" event={"ID":"545e1736-efb2-4c41-92fb-63195bd37784","Type":"ContainerDied","Data":"96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff"} Sep 09 17:39:11.883643 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.883574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp" event={"ID":"545e1736-efb2-4c41-92fb-63195bd37784","Type":"ContainerDied","Data":"0284e1acede7b87a0492bb77d659ac815693ded906586154c18328acb8cb279e"} Sep 09 17:39:11.883643 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.883599 2580 scope.go:117] "RemoveContainer" containerID="96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff" Sep 09 17:39:11.892969 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.892953 2580 scope.go:117] "RemoveContainer" containerID="96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff" Sep 09 17:39:11.893219 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:11.893201 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff\": container with ID starting with 96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff not found: ID does not exist" containerID="96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff" Sep 09 17:39:11.893270 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.893226 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff"} err="failed to get container status \"96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff\": rpc error: code = NotFound desc = could not find container \"96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff\": container with ID starting with 96b972cc314ace3f68baa4bf8879a5712b627dbc81ff461df65286cfcbc953ff not found: ID does not exist" Sep 09 17:39:11.904623 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.904601 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp"] Sep 09 17:39:11.908283 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.908260 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-12bac2-openshift-default-5644f47b46-5fpwp"] Sep 09 17:39:11.925491 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.925468 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx45k\" (UniqueName: \"kubernetes.io/projected/f150c868-970d-4322-b70b-8115200d69fb-kube-api-access-mx45k\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.925558 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.925522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f150c868-970d-4322-b70b-8115200d69fb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.925602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.925557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.925602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.925581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.925693 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:11.925679 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-serving-cert: secret "maas-api-e2e-shared-a-w3-480517-serving-cert" not found Sep 09 17:39:11.925768 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:11.925759 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls podName:f150c868-970d-4322-b70b-8115200d69fb nodeName:}" failed. No retries permitted until 2026-09-09 17:39:12.425744731 +0000 UTC m=+1346.774106187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls") pod "maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" (UID: "f150c868-970d-4322-b70b-8115200d69fb") : secret "maas-api-e2e-shared-a-w3-480517-serving-cert" not found Sep 09 17:39:11.926263 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.926247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f150c868-970d-4322-b70b-8115200d69fb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.927794 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.927778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:11.935994 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:11.935973 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx45k\" (UniqueName: \"kubernetes.io/projected/f150c868-970d-4322-b70b-8115200d69fb-kube-api-access-mx45k\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:12.133736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.133655 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545e1736-efb2-4c41-92fb-63195bd37784" path="/var/lib/kubelet/pods/545e1736-efb2-4c41-92fb-63195bd37784/volumes" Sep 09 17:39:12.273151 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.273127 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:39:12.429740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429632 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-envoy\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.429740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429667 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istiod-ca-cert\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.429740 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429691 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-credential-socket\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.430005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429839 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-token\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.430005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429868 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-data\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.430005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429903 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-socket\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.430005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429938 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-certs\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.430005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-podinfo\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.430005 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.429999 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-455z7\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-kube-api-access-455z7\") pod \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\" (UID: \"2e21e9a6-f10d-4cae-984c-4c681b6b0452\") " Sep 09 17:39:12.430315 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.430200 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:12.430315 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.430243 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:12.430435 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.430416 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-data\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.430656 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.430629 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:12.430656 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.430647 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:12.430923 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.430782 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:12.431093 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.431058 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:39:12.432228 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.432168 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:39:12.432895 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.432859 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 09 17:39:12.433101 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.433081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:12.433189 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.433099 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:39:12.433250 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.433209 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-kube-api-access-455z7" (OuterVolumeSpecName: "kube-api-access-455z7") pod "2e21e9a6-f10d-4cae-984c-4c681b6b0452" (UID: "2e21e9a6-f10d-4cae-984c-4c681b6b0452"). InnerVolumeSpecName "kube-api-access-455z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:39:12.531004 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.530965 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-token\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.531004 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.530994 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.531004 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.531004 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-workload-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.531004 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.531012 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-podinfo\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.531595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.531019 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-455z7\" (UniqueName: \"kubernetes.io/projected/2e21e9a6-f10d-4cae-984c-4c681b6b0452-kube-api-access-455z7\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.531595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.531029 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istio-envoy\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.531595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.531036 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e21e9a6-f10d-4cae-984c-4c681b6b0452-istiod-ca-cert\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.531595 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.531044 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e21e9a6-f10d-4cae-984c-4c681b6b0452-credential-socket\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:39:12.655155 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.655124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:12.891396 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.891356 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerID="da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1" exitCode=0 Sep 09 17:39:12.891555 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.891402 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" Sep 09 17:39:12.891555 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.891436 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" event={"ID":"2e21e9a6-f10d-4cae-984c-4c681b6b0452","Type":"ContainerDied","Data":"da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1"} Sep 09 17:39:12.891555 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.891485 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn" event={"ID":"2e21e9a6-f10d-4cae-984c-4c681b6b0452","Type":"ContainerDied","Data":"0d53ef33222c9f078627d104e75d58267d29ff8b029fd9a36e6794ebd2e745d0"} Sep 09 17:39:12.891555 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.891519 2580 scope.go:117] "RemoveContainer" containerID="da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1" Sep 09 17:39:12.900804 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.900786 2580 scope.go:117] "RemoveContainer" containerID="5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9" Sep 09 17:39:12.909465 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.909411 2580 scope.go:117] "RemoveContainer" containerID="da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1" Sep 09 17:39:12.909796 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:12.909750 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1\": container with ID starting with da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1 not found: ID does not exist" containerID="da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1" Sep 09 17:39:12.909796 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.909787 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1"} err="failed to get container status \"da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1\": rpc error: code = NotFound desc = could not find container \"da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1\": container with ID starting with da7865b19bd85eecad1711249adfff8d2dbdf8b7cded94f571418e2a078ecfb1 not found: ID does not exist" Sep 09 17:39:12.910066 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.909809 2580 scope.go:117] "RemoveContainer" containerID="5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9" Sep 09 17:39:12.910151 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:12.910112 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9\": container with ID starting with 5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9 not found: ID does not exist" containerID="5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9" Sep 09 17:39:12.910225 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.910144 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9"} err="failed to get container status \"5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9\": rpc error: code = NotFound desc = could not find container \"5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9\": container with ID starting with 5b1810c231d692a4ff68c81ec2b74af5ea17a89f3decafdb963632936e9ba5b9 not found: ID does not exist" Sep 09 17:39:12.912342 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.912321 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn"] Sep 09 17:39:12.915760 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.915733 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-53da12-openshift-default-67845b446d-9x4jn"] Sep 09 17:39:12.926329 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.926312 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6"] Sep 09 17:39:12.928233 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:39:12.928213 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf150c868_970d_4322_b70b_8115200d69fb.slice/crio-318397b59c6a2f0ca8c054d5ede0b576ab4b32012f1a765d9dafd34839ed60c6 WatchSource:0}: Error finding container 318397b59c6a2f0ca8c054d5ede0b576ab4b32012f1a765d9dafd34839ed60c6: Status 404 returned error can't find the container with id 318397b59c6a2f0ca8c054d5ede0b576ab4b32012f1a765d9dafd34839ed60c6 Sep 09 17:39:12.929420 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:12.929405 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:39:13.899762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:13.899729 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" event={"ID":"f150c868-970d-4322-b70b-8115200d69fb","Type":"ContainerStarted","Data":"1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1"} Sep 09 17:39:13.900220 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:13.899776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" event={"ID":"f150c868-970d-4322-b70b-8115200d69fb","Type":"ContainerStarted","Data":"318397b59c6a2f0ca8c054d5ede0b576ab4b32012f1a765d9dafd34839ed60c6"} Sep 09 17:39:13.900220 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:13.899806 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:13.919006 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:13.918922 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" podStartSLOduration=2.187981694 podStartE2EDuration="2.918905969s" totalImagesPullingTime="730.924275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:13.916535237 +0000 UTC m=+1348.264896708" watchObservedRunningTime="2026-09-09 17:39:13.918905969 +0000 UTC m=+1348.267267441" Sep 09 17:39:14.131209 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:14.131173 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" path="/var/lib/kubelet/pods/2e21e9a6-f10d-4cae-984c-4c681b6b0452/volumes" Sep 09 17:39:15.090874 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:15.090843 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:39:19.910686 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:19.910657 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:39:26.849062 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.849019 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c"] Sep 09 17:39:26.849505 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.849492 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" Sep 09 17:39:26.849552 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.849507 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" Sep 09 17:39:26.849599 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.849590 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" Sep 09 17:39:26.852794 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.852777 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:26.855267 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.855242 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-6b44e3-serving-cert\"" Sep 09 17:39:26.863530 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.863503 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c"] Sep 09 17:39:26.969101 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.969069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa76aa3-9f57-4b09-80c2-23c3185327b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:26.969270 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.969216 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:26.969270 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.969268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:26.969382 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:26.969285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmq5q\" (UniqueName: \"kubernetes.io/projected/0aa76aa3-9f57-4b09-80c2-23c3185327b8-kube-api-access-qmq5q\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.070106 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.070076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.070307 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.070122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.070307 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.070142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmq5q\" (UniqueName: \"kubernetes.io/projected/0aa76aa3-9f57-4b09-80c2-23c3185327b8-kube-api-access-qmq5q\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.070307 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:27.070181 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-serving-cert: secret "maas-api-e2e-shared-a-w2-6b44e3-serving-cert" not found Sep 09 17:39:27.070307 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.070184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa76aa3-9f57-4b09-80c2-23c3185327b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.070307 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:27.070241 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls podName:0aa76aa3-9f57-4b09-80c2-23c3185327b8 nodeName:}" failed. No retries permitted until 2026-09-09 17:39:27.570223141 +0000 UTC m=+1361.918584595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls") pod "maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" (UID: "0aa76aa3-9f57-4b09-80c2-23c3185327b8") : secret "maas-api-e2e-shared-a-w2-6b44e3-serving-cert" not found Sep 09 17:39:27.070835 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.070814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa76aa3-9f57-4b09-80c2-23c3185327b8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.072519 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.072498 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.078396 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.078370 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmq5q\" (UniqueName: \"kubernetes.io/projected/0aa76aa3-9f57-4b09-80c2-23c3185327b8-kube-api-access-qmq5q\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.575807 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.575765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.578046 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.578025 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:27.765399 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:27.765367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:28.036147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:28.036122 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c"] Sep 09 17:39:28.038250 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:39:28.038224 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa76aa3_9f57_4b09_80c2_23c3185327b8.slice/crio-a28ffd52a88afd90bf8504df324f5cae8ed10661044aa2b75a3fae276b31b280 WatchSource:0}: Error finding container a28ffd52a88afd90bf8504df324f5cae8ed10661044aa2b75a3fae276b31b280: Status 404 returned error can't find the container with id a28ffd52a88afd90bf8504df324f5cae8ed10661044aa2b75a3fae276b31b280 Sep 09 17:39:28.987242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:28.987211 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" event={"ID":"0aa76aa3-9f57-4b09-80c2-23c3185327b8","Type":"ContainerStarted","Data":"00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7"} Sep 09 17:39:28.987413 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:28.987262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" event={"ID":"0aa76aa3-9f57-4b09-80c2-23c3185327b8","Type":"ContainerStarted","Data":"a28ffd52a88afd90bf8504df324f5cae8ed10661044aa2b75a3fae276b31b280"} Sep 09 17:39:28.987413 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:28.987287 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:29.005812 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:29.005773 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" podStartSLOduration=2.588167123 podStartE2EDuration="3.005759538s" totalImagesPullingTime="417.592415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:29.002736387 +0000 UTC m=+1363.351097866" watchObservedRunningTime="2026-09-09 17:39:29.005759538 +0000 UTC m=+1363.354121012" Sep 09 17:39:30.592127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:30.592089 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:39:34.998227 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:34.998193 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:39:39.771365 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.771331 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf"] Sep 09 17:39:39.771872 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.771808 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" Sep 09 17:39:39.771872 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.771822 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" Sep 09 17:39:39.772011 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.771907 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e21e9a6-f10d-4cae-984c-4c681b6b0452" containerName="istio-proxy" Sep 09 17:39:39.775402 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.775355 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.778566 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.778541 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-17e67b-serving-cert\"" Sep 09 17:39:39.789317 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.789294 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf"] Sep 09 17:39:39.883532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.883495 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm8df\" (UniqueName: \"kubernetes.io/projected/5d9209df-cd54-467e-9e6a-a1b388c80096-kube-api-access-nm8df\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.883656 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.883546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9209df-cd54-467e-9e6a-a1b388c80096-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.883656 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.883602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.883758 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.883664 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.984367 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.984344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.984532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.984409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm8df\" (UniqueName: \"kubernetes.io/projected/5d9209df-cd54-467e-9e6a-a1b388c80096-kube-api-access-nm8df\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.984532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.984438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9209df-cd54-467e-9e6a-a1b388c80096-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.984532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.984462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.985167 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.985144 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9209df-cd54-467e-9e6a-a1b388c80096-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.986850 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.986827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.986931 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.986853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:39.992049 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:39.992021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm8df\" (UniqueName: \"kubernetes.io/projected/5d9209df-cd54-467e-9e6a-a1b388c80096-kube-api-access-nm8df\") pod \"maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:40.088363 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.088278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:40.366676 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.366649 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf"] Sep 09 17:39:40.368599 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:39:40.368572 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9209df_cd54_467e_9e6a_a1b388c80096.slice/crio-6f34fe748691e0ca58a88fae1f526161be778ec5707f68638f409b1fc56e765c WatchSource:0}: Error finding container 6f34fe748691e0ca58a88fae1f526161be778ec5707f68638f409b1fc56e765c: Status 404 returned error can't find the container with id 6f34fe748691e0ca58a88fae1f526161be778ec5707f68638f409b1fc56e765c Sep 09 17:39:40.418244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.418210 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv"] Sep 09 17:39:40.423014 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.422996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:40.425780 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.425758 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-17e67b\"" Sep 09 17:39:40.425937 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.425783 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-17e67b-dockercfg-tbb6k\"" Sep 09 17:39:40.427535 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.427515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv"] Sep 09 17:39:40.489281 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.489248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bzhg\" (UniqueName: \"kubernetes.io/projected/f71cf611-2a06-4496-8220-9081084a97ba-kube-api-access-8bzhg\") pod \"payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv\" (UID: \"f71cf611-2a06-4496-8220-9081084a97ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:40.489441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.489352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71cf611-2a06-4496-8220-9081084a97ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv\" (UID: \"f71cf611-2a06-4496-8220-9081084a97ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:40.590601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.590568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71cf611-2a06-4496-8220-9081084a97ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv\" (UID: \"f71cf611-2a06-4496-8220-9081084a97ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:40.590784 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.590646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bzhg\" (UniqueName: \"kubernetes.io/projected/f71cf611-2a06-4496-8220-9081084a97ba-kube-api-access-8bzhg\") pod \"payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv\" (UID: \"f71cf611-2a06-4496-8220-9081084a97ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:40.591263 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.591245 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71cf611-2a06-4496-8220-9081084a97ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv\" (UID: \"f71cf611-2a06-4496-8220-9081084a97ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:40.598955 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.598936 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bzhg\" (UniqueName: \"kubernetes.io/projected/f71cf611-2a06-4496-8220-9081084a97ba-kube-api-access-8bzhg\") pod \"payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv\" (UID: \"f71cf611-2a06-4496-8220-9081084a97ba\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:40.735872 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:40.735789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:41.006409 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:41.006372 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv"] Sep 09 17:39:41.007906 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:39:41.007881 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71cf611_2a06_4496_8220_9081084a97ba.slice/crio-b61771640ea6a9221b7feff97cfb96f301f63c73ada309eb8842b68058b1539d WatchSource:0}: Error finding container b61771640ea6a9221b7feff97cfb96f301f63c73ada309eb8842b68058b1539d: Status 404 returned error can't find the container with id b61771640ea6a9221b7feff97cfb96f301f63c73ada309eb8842b68058b1539d Sep 09 17:39:41.069597 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:41.069567 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" event={"ID":"f71cf611-2a06-4496-8220-9081084a97ba","Type":"ContainerStarted","Data":"b61771640ea6a9221b7feff97cfb96f301f63c73ada309eb8842b68058b1539d"} Sep 09 17:39:41.071403 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:41.071380 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" event={"ID":"5d9209df-cd54-467e-9e6a-a1b388c80096","Type":"ContainerStarted","Data":"cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6"} Sep 09 17:39:41.071508 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:41.071424 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" event={"ID":"5d9209df-cd54-467e-9e6a-a1b388c80096","Type":"ContainerStarted","Data":"6f34fe748691e0ca58a88fae1f526161be778ec5707f68638f409b1fc56e765c"} Sep 09 17:39:41.071508 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:41.071457 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:41.087246 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:41.087204 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" podStartSLOduration=1.550706631 podStartE2EDuration="2.087186849s" totalImagesPullingTime="536.480218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:41.085785331 +0000 UTC m=+1375.434146800" watchObservedRunningTime="2026-09-09 17:39:41.087186849 +0000 UTC m=+1375.435548322" Sep 09 17:39:42.079087 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:42.079057 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" event={"ID":"f71cf611-2a06-4496-8220-9081084a97ba","Type":"ContainerStarted","Data":"c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35"} Sep 09 17:39:42.079443 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:42.079309 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:42.094222 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:42.094182 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" podStartSLOduration=2.094168051 podStartE2EDuration="2.094168051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:42.092727217 +0000 UTC m=+1376.441088682" watchObservedRunningTime="2026-09-09 17:39:42.094168051 +0000 UTC m=+1376.442529523" Sep 09 17:39:43.881269 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:43.881232 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:39:47.085097 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:47.085068 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:39:53.086942 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:53.086913 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:39:55.042591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.042558 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc"] Sep 09 17:39:55.047626 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.047598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.050314 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.050290 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-607751-serving-cert\"" Sep 09 17:39:55.055775 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.055751 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc"] Sep 09 17:39:55.116517 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.116490 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.116632 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.116543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cznh\" (UniqueName: \"kubernetes.io/projected/e68411a2-7758-427b-973c-d0b04f3175b3-kube-api-access-2cznh\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.116632 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.116587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68411a2-7758-427b-973c-d0b04f3175b3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.116706 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.116688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.217497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.217465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.217654 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.217521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cznh\" (UniqueName: \"kubernetes.io/projected/e68411a2-7758-427b-973c-d0b04f3175b3-kube-api-access-2cznh\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.217654 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.217552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68411a2-7758-427b-973c-d0b04f3175b3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.217654 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:55.217598 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-serving-cert: secret "maas-api-e2e-shared-b-w2-607751-serving-cert" not found Sep 09 17:39:55.217818 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:39:55.217666 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls podName:e68411a2-7758-427b-973c-d0b04f3175b3 nodeName:}" failed. No retries permitted until 2026-09-09 17:39:55.717649572 +0000 UTC m=+1390.066011027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls") pod "maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" (UID: "e68411a2-7758-427b-973c-d0b04f3175b3") : secret "maas-api-e2e-shared-b-w2-607751-serving-cert" not found Sep 09 17:39:55.217818 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.217599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.218188 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.218167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68411a2-7758-427b-973c-d0b04f3175b3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.219951 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.219932 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.228187 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.228168 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cznh\" (UniqueName: \"kubernetes.io/projected/e68411a2-7758-427b-973c-d0b04f3175b3-kube-api-access-2cznh\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.723305 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.723261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.725560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.725537 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:55.961443 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:55.961411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:56.230486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:56.230444 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc"] Sep 09 17:39:56.238957 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:39:56.238928 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68411a2_7758_427b_973c_d0b04f3175b3.slice/crio-c9212d2e9b8b6e907d8ae62f42349a46694c58472c9965f561119c8e855c8ec0 WatchSource:0}: Error finding container c9212d2e9b8b6e907d8ae62f42349a46694c58472c9965f561119c8e855c8ec0: Status 404 returned error can't find the container with id c9212d2e9b8b6e907d8ae62f42349a46694c58472c9965f561119c8e855c8ec0 Sep 09 17:39:57.164978 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:57.164935 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" event={"ID":"e68411a2-7758-427b-973c-d0b04f3175b3","Type":"ContainerStarted","Data":"075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604"} Sep 09 17:39:57.165178 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:57.165081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:39:57.165178 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:57.165105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" event={"ID":"e68411a2-7758-427b-973c-d0b04f3175b3","Type":"ContainerStarted","Data":"c9212d2e9b8b6e907d8ae62f42349a46694c58472c9965f561119c8e855c8ec0"} Sep 09 17:39:57.181886 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:39:57.181839 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" podStartSLOduration=1.809809928 podStartE2EDuration="2.181824594s" totalImagesPullingTime="372.014666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:39:57.180415032 +0000 UTC m=+1391.528776506" watchObservedRunningTime="2026-09-09 17:39:57.181824594 +0000 UTC m=+1391.530186078" Sep 09 17:40:01.386170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:01.386136 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:40:03.176026 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:03.175998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:40:15.348625 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.348590 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6"] Sep 09 17:40:15.349212 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.348854 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" podUID="f150c868-970d-4322-b70b-8115200d69fb" containerName="maas-api" containerID="cri-o://1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1" gracePeriod=30 Sep 09 17:40:15.595065 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.595038 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:40:15.605552 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.605501 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls\") pod \"f150c868-970d-4322-b70b-8115200d69fb\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " Sep 09 17:40:15.605552 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.605532 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-metrics-certs\") pod \"f150c868-970d-4322-b70b-8115200d69fb\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " Sep 09 17:40:15.605675 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.605558 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx45k\" (UniqueName: \"kubernetes.io/projected/f150c868-970d-4322-b70b-8115200d69fb-kube-api-access-mx45k\") pod \"f150c868-970d-4322-b70b-8115200d69fb\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " Sep 09 17:40:15.605675 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.605582 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f150c868-970d-4322-b70b-8115200d69fb-openshift-service-ca\") pod \"f150c868-970d-4322-b70b-8115200d69fb\" (UID: \"f150c868-970d-4322-b70b-8115200d69fb\") " Sep 09 17:40:15.606020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.605997 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f150c868-970d-4322-b70b-8115200d69fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f150c868-970d-4322-b70b-8115200d69fb" (UID: "f150c868-970d-4322-b70b-8115200d69fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:40:15.607608 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.607578 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f150c868-970d-4322-b70b-8115200d69fb" (UID: "f150c868-970d-4322-b70b-8115200d69fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:40:15.607691 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.607635 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f150c868-970d-4322-b70b-8115200d69fb-kube-api-access-mx45k" (OuterVolumeSpecName: "kube-api-access-mx45k") pod "f150c868-970d-4322-b70b-8115200d69fb" (UID: "f150c868-970d-4322-b70b-8115200d69fb"). InnerVolumeSpecName "kube-api-access-mx45k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:40:15.607756 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.607703 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f150c868-970d-4322-b70b-8115200d69fb" (UID: "f150c868-970d-4322-b70b-8115200d69fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:40:15.706542 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.706505 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:15.706542 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.706531 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f150c868-970d-4322-b70b-8115200d69fb-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:15.706542 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.706541 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx45k\" (UniqueName: \"kubernetes.io/projected/f150c868-970d-4322-b70b-8115200d69fb-kube-api-access-mx45k\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:15.706857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:15.706549 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f150c868-970d-4322-b70b-8115200d69fb-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:16.272666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.272634 2580 generic.go:356] "Generic (PLEG): container finished" podID="f150c868-970d-4322-b70b-8115200d69fb" containerID="1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1" exitCode=0 Sep 09 17:40:16.272857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.272677 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" Sep 09 17:40:16.272857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.272744 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" event={"ID":"f150c868-970d-4322-b70b-8115200d69fb","Type":"ContainerDied","Data":"1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1"} Sep 09 17:40:16.272857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.272799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6" event={"ID":"f150c868-970d-4322-b70b-8115200d69fb","Type":"ContainerDied","Data":"318397b59c6a2f0ca8c054d5ede0b576ab4b32012f1a765d9dafd34839ed60c6"} Sep 09 17:40:16.272857 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.272826 2580 scope.go:117] "RemoveContainer" containerID="1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1" Sep 09 17:40:16.290328 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.290304 2580 scope.go:117] "RemoveContainer" containerID="1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1" Sep 09 17:40:16.290878 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:40:16.290827 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1\": container with ID starting with 1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1 not found: ID does not exist" containerID="1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1" Sep 09 17:40:16.290992 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.290968 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1"} err="failed to get container status \"1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1\": rpc error: code = NotFound desc = could not find container \"1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1\": container with ID starting with 1198f4faed05c368c7fdeac1db631b77218fb0396b01167de15919cc2197fbb1 not found: ID does not exist" Sep 09 17:40:16.298422 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.298397 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6"] Sep 09 17:40:16.300266 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:16.300244 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-480517-786795d976-7w4m6"] Sep 09 17:40:18.131996 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:18.131964 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f150c868-970d-4322-b70b-8115200d69fb" path="/var/lib/kubelet/pods/f150c868-970d-4322-b70b-8115200d69fb/volumes" Sep 09 17:40:18.789590 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:18.789551 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:40:19.419345 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.419314 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:19.419778 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.419765 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f150c868-970d-4322-b70b-8115200d69fb" containerName="maas-api" Sep 09 17:40:19.419834 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.419779 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f150c868-970d-4322-b70b-8115200d69fb" containerName="maas-api" Sep 09 17:40:19.419869 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.419862 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f150c868-970d-4322-b70b-8115200d69fb" containerName="maas-api" Sep 09 17:40:19.423214 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.423199 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:19.426004 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.425981 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p6jf9\"" Sep 09 17:40:19.429229 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.429205 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:19.438207 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.438182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zqmc\" (UniqueName: \"kubernetes.io/projected/92d2052d-7d64-42f9-a801-0f16af4b134c-kube-api-access-7zqmc\") pod \"test-curl-7664-140108860405776\" (UID: \"92d2052d-7d64-42f9-a801-0f16af4b134c\") " pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:19.539388 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.539360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zqmc\" (UniqueName: \"kubernetes.io/projected/92d2052d-7d64-42f9-a801-0f16af4b134c-kube-api-access-7zqmc\") pod \"test-curl-7664-140108860405776\" (UID: \"92d2052d-7d64-42f9-a801-0f16af4b134c\") " pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:19.547754 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.547702 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zqmc\" (UniqueName: \"kubernetes.io/projected/92d2052d-7d64-42f9-a801-0f16af4b134c-kube-api-access-7zqmc\") pod \"test-curl-7664-140108860405776\" (UID: \"92d2052d-7d64-42f9-a801-0f16af4b134c\") " pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:19.734830 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:19.734746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:20.001970 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:20.001945 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:20.003629 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:40:20.003602 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92d2052d_7d64_42f9_a801_0f16af4b134c.slice/crio-a2e3860ebf6d1ac25cd9f7e5c6b3ce11f1bb68a6a75220d18bbd43c8cd3ac0fb WatchSource:0}: Error finding container a2e3860ebf6d1ac25cd9f7e5c6b3ce11f1bb68a6a75220d18bbd43c8cd3ac0fb: Status 404 returned error can't find the container with id a2e3860ebf6d1ac25cd9f7e5c6b3ce11f1bb68a6a75220d18bbd43c8cd3ac0fb Sep 09 17:40:20.297311 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:20.297278 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140108860405776" event={"ID":"92d2052d-7d64-42f9-a801-0f16af4b134c","Type":"ContainerStarted","Data":"a2e3860ebf6d1ac25cd9f7e5c6b3ce11f1bb68a6a75220d18bbd43c8cd3ac0fb"} Sep 09 17:40:21.304003 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:21.303966 2580 generic.go:356] "Generic (PLEG): container finished" podID="92d2052d-7d64-42f9-a801-0f16af4b134c" containerID="bf2127ff2f11cd323fafc58d13819c1872aa1a46a638000878a4e80ce1cd7d38" exitCode=0 Sep 09 17:40:21.304388 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:21.304054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140108860405776" event={"ID":"92d2052d-7d64-42f9-a801-0f16af4b134c","Type":"ContainerDied","Data":"bf2127ff2f11cd323fafc58d13819c1872aa1a46a638000878a4e80ce1cd7d38"} Sep 09 17:40:22.450449 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.450426 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:22.484825 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.484802 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140108860405776_92d2052d-7d64-42f9-a801-0f16af4b134c/test-curl-7664-140108860405776/0.log" Sep 09 17:40:22.508559 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.508538 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:22.512474 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.512450 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:22.567486 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.567420 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zqmc\" (UniqueName: \"kubernetes.io/projected/92d2052d-7d64-42f9-a801-0f16af4b134c-kube-api-access-7zqmc\") pod \"92d2052d-7d64-42f9-a801-0f16af4b134c\" (UID: \"92d2052d-7d64-42f9-a801-0f16af4b134c\") " Sep 09 17:40:22.569506 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.569480 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d2052d-7d64-42f9-a801-0f16af4b134c-kube-api-access-7zqmc" (OuterVolumeSpecName: "kube-api-access-7zqmc") pod "92d2052d-7d64-42f9-a801-0f16af4b134c" (UID: "92d2052d-7d64-42f9-a801-0f16af4b134c"). InnerVolumeSpecName "kube-api-access-7zqmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:40:22.629621 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.629598 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:22.630127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.630110 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d2052d-7d64-42f9-a801-0f16af4b134c" containerName="test-curl-7664-140108860405776" Sep 09 17:40:22.630127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.630128 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d2052d-7d64-42f9-a801-0f16af4b134c" containerName="test-curl-7664-140108860405776" Sep 09 17:40:22.630243 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.630220 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d2052d-7d64-42f9-a801-0f16af4b134c" containerName="test-curl-7664-140108860405776" Sep 09 17:40:22.633325 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.633310 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:22.639033 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.639010 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:22.668407 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.668382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8p2\" (UniqueName: \"kubernetes.io/projected/2e358b2e-da60-444c-acd1-695b2b7b357c-kube-api-access-lx8p2\") pod \"test-curl-7664-140108860405776\" (UID: \"2e358b2e-da60-444c-acd1-695b2b7b357c\") " pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:22.668567 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.668550 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zqmc\" (UniqueName: \"kubernetes.io/projected/92d2052d-7d64-42f9-a801-0f16af4b134c-kube-api-access-7zqmc\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:22.769169 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.769144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8p2\" (UniqueName: \"kubernetes.io/projected/2e358b2e-da60-444c-acd1-695b2b7b357c-kube-api-access-lx8p2\") pod \"test-curl-7664-140108860405776\" (UID: \"2e358b2e-da60-444c-acd1-695b2b7b357c\") " pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:22.776815 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.776793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8p2\" (UniqueName: \"kubernetes.io/projected/2e358b2e-da60-444c-acd1-695b2b7b357c-kube-api-access-lx8p2\") pod \"test-curl-7664-140108860405776\" (UID: \"2e358b2e-da60-444c-acd1-695b2b7b357c\") " pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:22.945513 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:22.945443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:23.212581 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:23.212556 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:23.214414 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:40:23.214388 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e358b2e_da60_444c_acd1_695b2b7b357c.slice/crio-6985c44488237bc5dae4425349e95296f08525595c128358c834904928b5a5df WatchSource:0}: Error finding container 6985c44488237bc5dae4425349e95296f08525595c128358c834904928b5a5df: Status 404 returned error can't find the container with id 6985c44488237bc5dae4425349e95296f08525595c128358c834904928b5a5df Sep 09 17:40:23.316408 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:23.316381 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140108860405776" event={"ID":"2e358b2e-da60-444c-acd1-695b2b7b357c","Type":"ContainerStarted","Data":"6985c44488237bc5dae4425349e95296f08525595c128358c834904928b5a5df"} Sep 09 17:40:23.317633 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:23.317615 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e3860ebf6d1ac25cd9f7e5c6b3ce11f1bb68a6a75220d18bbd43c8cd3ac0fb" Sep 09 17:40:23.317633 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:23.317623 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:23.328823 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:23.328798 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7664-140108860405776" oldPodUID="92d2052d-7d64-42f9-a801-0f16af4b134c" podUID="2e358b2e-da60-444c-acd1-695b2b7b357c" Sep 09 17:40:24.131468 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:24.131441 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d2052d-7d64-42f9-a801-0f16af4b134c" path="/var/lib/kubelet/pods/92d2052d-7d64-42f9-a801-0f16af4b134c/volumes" Sep 09 17:40:24.324177 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:24.324144 2580 generic.go:356] "Generic (PLEG): container finished" podID="2e358b2e-da60-444c-acd1-695b2b7b357c" containerID="67245cd83c74883a42c1f46c30ead5aeba9eec84b4e2554e4d437ac13b8c564e" exitCode=0 Sep 09 17:40:24.324319 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:24.324229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140108860405776" event={"ID":"2e358b2e-da60-444c-acd1-695b2b7b357c","Type":"ContainerDied","Data":"67245cd83c74883a42c1f46c30ead5aeba9eec84b4e2554e4d437ac13b8c564e"} Sep 09 17:40:25.465029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.465006 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:25.484965 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.484937 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140108860405776_2e358b2e-da60-444c-acd1-695b2b7b357c/test-curl-7664-140108860405776/0.log" Sep 09 17:40:25.496343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.496319 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx8p2\" (UniqueName: \"kubernetes.io/projected/2e358b2e-da60-444c-acd1-695b2b7b357c-kube-api-access-lx8p2\") pod \"2e358b2e-da60-444c-acd1-695b2b7b357c\" (UID: \"2e358b2e-da60-444c-acd1-695b2b7b357c\") " Sep 09 17:40:25.498230 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.498206 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e358b2e-da60-444c-acd1-695b2b7b357c-kube-api-access-lx8p2" (OuterVolumeSpecName: "kube-api-access-lx8p2") pod "2e358b2e-da60-444c-acd1-695b2b7b357c" (UID: "2e358b2e-da60-444c-acd1-695b2b7b357c"). InnerVolumeSpecName "kube-api-access-lx8p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:40:25.507668 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.507646 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:25.510845 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.510825 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140108860405776"] Sep 09 17:40:25.597627 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.597571 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8p2\" (UniqueName: \"kubernetes.io/projected/2e358b2e-da60-444c-acd1-695b2b7b357c-kube-api-access-lx8p2\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:25.623133 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.623109 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:25.623590 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.623576 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e358b2e-da60-444c-acd1-695b2b7b357c" containerName="test-curl-7664-140108860405776" Sep 09 17:40:25.623636 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.623592 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e358b2e-da60-444c-acd1-695b2b7b357c" containerName="test-curl-7664-140108860405776" Sep 09 17:40:25.623671 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.623665 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e358b2e-da60-444c-acd1-695b2b7b357c" containerName="test-curl-7664-140108860405776" Sep 09 17:40:25.626667 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.626653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:25.632395 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.632375 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:25.698450 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.698422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtsh2\" (UniqueName: \"kubernetes.io/projected/559b899c-ee8e-43dd-a6e4-9320ede03196-kube-api-access-xtsh2\") pod \"test-curl-7664-140108860406352\" (UID: \"559b899c-ee8e-43dd-a6e4-9320ede03196\") " pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:25.799417 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.799392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtsh2\" (UniqueName: \"kubernetes.io/projected/559b899c-ee8e-43dd-a6e4-9320ede03196-kube-api-access-xtsh2\") pod \"test-curl-7664-140108860406352\" (UID: \"559b899c-ee8e-43dd-a6e4-9320ede03196\") " pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:25.809168 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.809147 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtsh2\" (UniqueName: \"kubernetes.io/projected/559b899c-ee8e-43dd-a6e4-9320ede03196-kube-api-access-xtsh2\") pod \"test-curl-7664-140108860406352\" (UID: \"559b899c-ee8e-43dd-a6e4-9320ede03196\") " pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:25.937677 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:25.937613 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:26.130934 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:26.130900 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e358b2e-da60-444c-acd1-695b2b7b357c" path="/var/lib/kubelet/pods/2e358b2e-da60-444c-acd1-695b2b7b357c/volumes" Sep 09 17:40:26.206036 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:26.206011 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:26.207840 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:40:26.207819 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod559b899c_ee8e_43dd_a6e4_9320ede03196.slice/crio-cc2273252fb7ef0eb4cd2cf7a614ece406be2b9d3cf18c85502a03bdd53aa7bf WatchSource:0}: Error finding container cc2273252fb7ef0eb4cd2cf7a614ece406be2b9d3cf18c85502a03bdd53aa7bf: Status 404 returned error can't find the container with id cc2273252fb7ef0eb4cd2cf7a614ece406be2b9d3cf18c85502a03bdd53aa7bf Sep 09 17:40:26.337478 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:26.337450 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860405776" Sep 09 17:40:26.337478 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:26.337467 2580 scope.go:117] "RemoveContainer" containerID="67245cd83c74883a42c1f46c30ead5aeba9eec84b4e2554e4d437ac13b8c564e" Sep 09 17:40:26.339126 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:26.338966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140108860406352" event={"ID":"559b899c-ee8e-43dd-a6e4-9320ede03196","Type":"ContainerStarted","Data":"cc2273252fb7ef0eb4cd2cf7a614ece406be2b9d3cf18c85502a03bdd53aa7bf"} Sep 09 17:40:27.347727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:27.347669 2580 generic.go:356] "Generic (PLEG): container finished" podID="559b899c-ee8e-43dd-a6e4-9320ede03196" containerID="17092a91b4af43985a0d4e4f436191975c6a4980ab2dfebd0580c97f8cd99b26" exitCode=0 Sep 09 17:40:27.348169 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:27.347751 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140108860406352" event={"ID":"559b899c-ee8e-43dd-a6e4-9320ede03196","Type":"ContainerDied","Data":"17092a91b4af43985a0d4e4f436191975c6a4980ab2dfebd0580c97f8cd99b26"} Sep 09 17:40:28.485533 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.485509 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:28.504182 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.504155 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140108860406352_559b899c-ee8e-43dd-a6e4-9320ede03196/test-curl-7664-140108860406352/0.log" Sep 09 17:40:28.523335 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.523308 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtsh2\" (UniqueName: \"kubernetes.io/projected/559b899c-ee8e-43dd-a6e4-9320ede03196-kube-api-access-xtsh2\") pod \"559b899c-ee8e-43dd-a6e4-9320ede03196\" (UID: \"559b899c-ee8e-43dd-a6e4-9320ede03196\") " Sep 09 17:40:28.525622 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.525561 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559b899c-ee8e-43dd-a6e4-9320ede03196-kube-api-access-xtsh2" (OuterVolumeSpecName: "kube-api-access-xtsh2") pod "559b899c-ee8e-43dd-a6e4-9320ede03196" (UID: "559b899c-ee8e-43dd-a6e4-9320ede03196"). InnerVolumeSpecName "kube-api-access-xtsh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:40:28.526928 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.526907 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:28.530152 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.530128 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:28.624108 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.624053 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtsh2\" (UniqueName: \"kubernetes.io/projected/559b899c-ee8e-43dd-a6e4-9320ede03196-kube-api-access-xtsh2\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:28.649289 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.649268 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:28.649786 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.649771 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="559b899c-ee8e-43dd-a6e4-9320ede03196" containerName="test-curl-7664-140108860406352" Sep 09 17:40:28.649829 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.649788 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="559b899c-ee8e-43dd-a6e4-9320ede03196" containerName="test-curl-7664-140108860406352" Sep 09 17:40:28.649872 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.649862 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="559b899c-ee8e-43dd-a6e4-9320ede03196" containerName="test-curl-7664-140108860406352" Sep 09 17:40:28.653025 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.653009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:28.659093 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.659069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:28.724540 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.724520 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2phw\" (UniqueName: \"kubernetes.io/projected/c6a3ee72-26e8-40c2-b1d9-46d3d31e044f-kube-api-access-t2phw\") pod \"test-curl-7664-140108860406352\" (UID: \"c6a3ee72-26e8-40c2-b1d9-46d3d31e044f\") " pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:28.825489 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.825458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2phw\" (UniqueName: \"kubernetes.io/projected/c6a3ee72-26e8-40c2-b1d9-46d3d31e044f-kube-api-access-t2phw\") pod \"test-curl-7664-140108860406352\" (UID: \"c6a3ee72-26e8-40c2-b1d9-46d3d31e044f\") " pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:28.833893 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.833864 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2phw\" (UniqueName: \"kubernetes.io/projected/c6a3ee72-26e8-40c2-b1d9-46d3d31e044f-kube-api-access-t2phw\") pod \"test-curl-7664-140108860406352\" (UID: \"c6a3ee72-26e8-40c2-b1d9-46d3d31e044f\") " pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:28.964354 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:28.964278 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:29.235883 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:29.235856 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:29.237985 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:40:29.237958 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6a3ee72_26e8_40c2_b1d9_46d3d31e044f.slice/crio-3bd40c572fb8c6258ddb7386638c459ae782b9e810c9dbe37b6098b5b833f0d7 WatchSource:0}: Error finding container 3bd40c572fb8c6258ddb7386638c459ae782b9e810c9dbe37b6098b5b833f0d7: Status 404 returned error can't find the container with id 3bd40c572fb8c6258ddb7386638c459ae782b9e810c9dbe37b6098b5b833f0d7 Sep 09 17:40:29.360169 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:29.360142 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc2273252fb7ef0eb4cd2cf7a614ece406be2b9d3cf18c85502a03bdd53aa7bf" Sep 09 17:40:29.360169 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:29.360145 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:29.361401 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:29.361376 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140108860406352" event={"ID":"c6a3ee72-26e8-40c2-b1d9-46d3d31e044f","Type":"ContainerStarted","Data":"3bd40c572fb8c6258ddb7386638c459ae782b9e810c9dbe37b6098b5b833f0d7"} Sep 09 17:40:29.371922 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:29.371894 2580 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7664-140108860406352" oldPodUID="559b899c-ee8e-43dd-a6e4-9320ede03196" podUID="c6a3ee72-26e8-40c2-b1d9-46d3d31e044f" Sep 09 17:40:30.130945 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:30.130914 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="559b899c-ee8e-43dd-a6e4-9320ede03196" path="/var/lib/kubelet/pods/559b899c-ee8e-43dd-a6e4-9320ede03196/volumes" Sep 09 17:40:30.369057 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:30.368978 2580 generic.go:356] "Generic (PLEG): container finished" podID="c6a3ee72-26e8-40c2-b1d9-46d3d31e044f" containerID="53bf2133329d79faa9c092a431ae850ff1085d96ee6573af1fbda14afc389eb6" exitCode=0 Sep 09 17:40:30.369203 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:30.369054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-140108860406352" event={"ID":"c6a3ee72-26e8-40c2-b1d9-46d3d31e044f","Type":"ContainerDied","Data":"53bf2133329d79faa9c092a431ae850ff1085d96ee6573af1fbda14afc389eb6"} Sep 09 17:40:31.505860 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:31.505833 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:31.524850 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:31.524823 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7664-140108860406352_c6a3ee72-26e8-40c2-b1d9-46d3d31e044f/test-curl-7664-140108860406352/0.log" Sep 09 17:40:31.550265 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:31.550233 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:31.550534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:31.550519 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2phw\" (UniqueName: \"kubernetes.io/projected/c6a3ee72-26e8-40c2-b1d9-46d3d31e044f-kube-api-access-t2phw\") pod \"c6a3ee72-26e8-40c2-b1d9-46d3d31e044f\" (UID: \"c6a3ee72-26e8-40c2-b1d9-46d3d31e044f\") " Sep 09 17:40:31.552182 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:31.552153 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-140108860406352"] Sep 09 17:40:31.552601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:31.552579 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a3ee72-26e8-40c2-b1d9-46d3d31e044f-kube-api-access-t2phw" (OuterVolumeSpecName: "kube-api-access-t2phw") pod "c6a3ee72-26e8-40c2-b1d9-46d3d31e044f" (UID: "c6a3ee72-26e8-40c2-b1d9-46d3d31e044f"). InnerVolumeSpecName "kube-api-access-t2phw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:40:31.652320 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:31.652240 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2phw\" (UniqueName: \"kubernetes.io/projected/c6a3ee72-26e8-40c2-b1d9-46d3d31e044f-kube-api-access-t2phw\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:32.131567 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.131532 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a3ee72-26e8-40c2-b1d9-46d3d31e044f" path="/var/lib/kubelet/pods/c6a3ee72-26e8-40c2-b1d9-46d3d31e044f/volumes" Sep 09 17:40:32.382727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.382623 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-140108860406352" Sep 09 17:40:32.382727 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.382663 2580 scope.go:117] "RemoveContainer" containerID="53bf2133329d79faa9c092a431ae850ff1085d96ee6573af1fbda14afc389eb6" Sep 09 17:40:32.651024 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.650946 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf"] Sep 09 17:40:32.651434 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.651192 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" podUID="5d9209df-cd54-467e-9e6a-a1b388c80096" containerName="maas-api" containerID="cri-o://cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6" gracePeriod=30 Sep 09 17:40:32.702736 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.702690 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv"] Sep 09 17:40:32.702970 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.702941 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" podUID="f71cf611-2a06-4496-8220-9081084a97ba" containerName="payload-processing" containerID="cri-o://c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35" gracePeriod=30 Sep 09 17:40:32.955612 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.955587 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:40:32.988069 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:32.988046 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:40:33.066567 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.066532 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71cf611-2a06-4496-8220-9081084a97ba-plugins-config-volume\") pod \"f71cf611-2a06-4496-8220-9081084a97ba\" (UID: \"f71cf611-2a06-4496-8220-9081084a97ba\") " Sep 09 17:40:33.066737 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.066605 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm8df\" (UniqueName: \"kubernetes.io/projected/5d9209df-cd54-467e-9e6a-a1b388c80096-kube-api-access-nm8df\") pod \"5d9209df-cd54-467e-9e6a-a1b388c80096\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " Sep 09 17:40:33.066737 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.066690 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-metrics-certs\") pod \"5d9209df-cd54-467e-9e6a-a1b388c80096\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " Sep 09 17:40:33.066818 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.066781 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bzhg\" (UniqueName: \"kubernetes.io/projected/f71cf611-2a06-4496-8220-9081084a97ba-kube-api-access-8bzhg\") pod \"f71cf611-2a06-4496-8220-9081084a97ba\" (UID: \"f71cf611-2a06-4496-8220-9081084a97ba\") " Sep 09 17:40:33.066866 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.066827 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9209df-cd54-467e-9e6a-a1b388c80096-openshift-service-ca\") pod \"5d9209df-cd54-467e-9e6a-a1b388c80096\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " Sep 09 17:40:33.066866 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.066849 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-maas-api-tls\") pod \"5d9209df-cd54-467e-9e6a-a1b388c80096\" (UID: \"5d9209df-cd54-467e-9e6a-a1b388c80096\") " Sep 09 17:40:33.066998 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.066975 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71cf611-2a06-4496-8220-9081084a97ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f71cf611-2a06-4496-8220-9081084a97ba" (UID: "f71cf611-2a06-4496-8220-9081084a97ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:40:33.067249 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.067215 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9209df-cd54-467e-9e6a-a1b388c80096-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d9209df-cd54-467e-9e6a-a1b388c80096" (UID: "5d9209df-cd54-467e-9e6a-a1b388c80096"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:40:33.067362 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.067305 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71cf611-2a06-4496-8220-9081084a97ba-plugins-config-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:33.068972 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.068945 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9209df-cd54-467e-9e6a-a1b388c80096-kube-api-access-nm8df" (OuterVolumeSpecName: "kube-api-access-nm8df") pod "5d9209df-cd54-467e-9e6a-a1b388c80096" (UID: "5d9209df-cd54-467e-9e6a-a1b388c80096"). InnerVolumeSpecName "kube-api-access-nm8df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:40:33.068972 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.068952 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71cf611-2a06-4496-8220-9081084a97ba-kube-api-access-8bzhg" (OuterVolumeSpecName: "kube-api-access-8bzhg") pod "f71cf611-2a06-4496-8220-9081084a97ba" (UID: "f71cf611-2a06-4496-8220-9081084a97ba"). InnerVolumeSpecName "kube-api-access-8bzhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:40:33.069088 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.068950 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5d9209df-cd54-467e-9e6a-a1b388c80096" (UID: "5d9209df-cd54-467e-9e6a-a1b388c80096"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:40:33.069088 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.068981 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d9209df-cd54-467e-9e6a-a1b388c80096" (UID: "5d9209df-cd54-467e-9e6a-a1b388c80096"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:40:33.168107 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.168006 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm8df\" (UniqueName: \"kubernetes.io/projected/5d9209df-cd54-467e-9e6a-a1b388c80096-kube-api-access-nm8df\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:33.168107 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.168045 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:33.168107 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.168058 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bzhg\" (UniqueName: \"kubernetes.io/projected/f71cf611-2a06-4496-8220-9081084a97ba-kube-api-access-8bzhg\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:33.168107 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.168071 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d9209df-cd54-467e-9e6a-a1b388c80096-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:33.168107 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.168082 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d9209df-cd54-467e-9e6a-a1b388c80096-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:40:33.390264 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.390229 2580 generic.go:356] "Generic (PLEG): container finished" podID="f71cf611-2a06-4496-8220-9081084a97ba" containerID="c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35" exitCode=0 Sep 09 17:40:33.390421 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.390282 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" Sep 09 17:40:33.390421 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.390322 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" event={"ID":"f71cf611-2a06-4496-8220-9081084a97ba","Type":"ContainerDied","Data":"c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35"} Sep 09 17:40:33.390421 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.390382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv" event={"ID":"f71cf611-2a06-4496-8220-9081084a97ba","Type":"ContainerDied","Data":"b61771640ea6a9221b7feff97cfb96f301f63c73ada309eb8842b68058b1539d"} Sep 09 17:40:33.390421 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.390411 2580 scope.go:117] "RemoveContainer" containerID="c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35" Sep 09 17:40:33.391800 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.391666 2580 generic.go:356] "Generic (PLEG): container finished" podID="5d9209df-cd54-467e-9e6a-a1b388c80096" containerID="cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6" exitCode=0 Sep 09 17:40:33.391800 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.391704 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" Sep 09 17:40:33.391800 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.391767 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" event={"ID":"5d9209df-cd54-467e-9e6a-a1b388c80096","Type":"ContainerDied","Data":"cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6"} Sep 09 17:40:33.391987 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.391817 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf" event={"ID":"5d9209df-cd54-467e-9e6a-a1b388c80096","Type":"ContainerDied","Data":"6f34fe748691e0ca58a88fae1f526161be778ec5707f68638f409b1fc56e765c"} Sep 09 17:40:33.400542 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.400527 2580 scope.go:117] "RemoveContainer" containerID="c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35" Sep 09 17:40:33.400968 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:40:33.400841 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35\": container with ID starting with c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35 not found: ID does not exist" containerID="c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35" Sep 09 17:40:33.400968 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.400862 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35"} err="failed to get container status \"c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35\": rpc error: code = NotFound desc = could not find container \"c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35\": container with ID starting with c18962f3c654eefaab079835d71a8ed36c54041bdcd1f4459a87626648064e35 not found: ID does not exist" Sep 09 17:40:33.400968 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.400878 2580 scope.go:117] "RemoveContainer" containerID="cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6" Sep 09 17:40:33.409536 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.409509 2580 scope.go:117] "RemoveContainer" containerID="cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6" Sep 09 17:40:33.409822 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:40:33.409805 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6\": container with ID starting with cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6 not found: ID does not exist" containerID="cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6" Sep 09 17:40:33.409883 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.409827 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6"} err="failed to get container status \"cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6\": rpc error: code = NotFound desc = could not find container \"cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6\": container with ID starting with cfee171d083b1c5cc65ad4df12921ac5aaa74a1b1ff3e972b89caf0e64be89a6 not found: ID does not exist" Sep 09 17:40:33.413170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.413151 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv"] Sep 09 17:40:33.416301 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.416279 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-17e67b-57dbbdd957-qvbxv"] Sep 09 17:40:33.428376 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.428320 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf"] Sep 09 17:40:33.432149 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:33.432128 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-17e67b-5574bc5f6d-ncpgf"] Sep 09 17:40:34.131875 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:34.131836 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d9209df-cd54-467e-9e6a-a1b388c80096" path="/var/lib/kubelet/pods/5d9209df-cd54-467e-9e6a-a1b388c80096/volumes" Sep 09 17:40:34.132440 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:34.132419 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71cf611-2a06-4496-8220-9081084a97ba" path="/var/lib/kubelet/pods/f71cf611-2a06-4496-8220-9081084a97ba/volumes" Sep 09 17:40:35.783852 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:35.783821 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:40:36.134035 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.133944 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb"] Sep 09 17:40:36.134588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134573 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9209df-cd54-467e-9e6a-a1b388c80096" containerName="maas-api" Sep 09 17:40:36.134668 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134591 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9209df-cd54-467e-9e6a-a1b388c80096" containerName="maas-api" Sep 09 17:40:36.134668 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134629 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71cf611-2a06-4496-8220-9081084a97ba" containerName="payload-processing" Sep 09 17:40:36.134668 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134638 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71cf611-2a06-4496-8220-9081084a97ba" containerName="payload-processing" Sep 09 17:40:36.134668 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134647 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a3ee72-26e8-40c2-b1d9-46d3d31e044f" containerName="test-curl-7664-140108860406352" Sep 09 17:40:36.134668 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134656 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a3ee72-26e8-40c2-b1d9-46d3d31e044f" containerName="test-curl-7664-140108860406352" Sep 09 17:40:36.134939 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134774 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a3ee72-26e8-40c2-b1d9-46d3d31e044f" containerName="test-curl-7664-140108860406352" Sep 09 17:40:36.134939 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134794 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9209df-cd54-467e-9e6a-a1b388c80096" containerName="maas-api" Sep 09 17:40:36.134939 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.134802 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71cf611-2a06-4496-8220-9081084a97ba" containerName="payload-processing" Sep 09 17:40:36.140482 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.140453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.140836 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.140811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb"] Sep 09 17:40:36.143076 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.143050 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f636a1c6-serving-cert\"" Sep 09 17:40:36.206443 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.206392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-metrics-certs\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.206619 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.206511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-maas-api-tls\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.206619 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.206578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9501e0-9c45-4918-bd7e-d09927651402-openshift-service-ca\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.206619 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.206609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgzp\" (UniqueName: \"kubernetes.io/projected/5c9501e0-9c45-4918-bd7e-d09927651402-kube-api-access-4pgzp\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.307749 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.307688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgzp\" (UniqueName: \"kubernetes.io/projected/5c9501e0-9c45-4918-bd7e-d09927651402-kube-api-access-4pgzp\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.307918 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.307783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-metrics-certs\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.307918 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.307843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-maas-api-tls\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.307918 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.307867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9501e0-9c45-4918-bd7e-d09927651402-openshift-service-ca\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.308518 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.308495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9501e0-9c45-4918-bd7e-d09927651402-openshift-service-ca\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.310336 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.310307 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-metrics-certs\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.310442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.310399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-maas-api-tls\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.316283 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.316257 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgzp\" (UniqueName: \"kubernetes.io/projected/5c9501e0-9c45-4918-bd7e-d09927651402-kube-api-access-4pgzp\") pod \"maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.453814 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.453730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:36.737143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.737111 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb"] Sep 09 17:40:36.739664 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:40:36.739630 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9501e0_9c45_4918_bd7e_d09927651402.slice/crio-b3b5b88f5ef860add6b6a00784b65374847adf5a2a7adf8aa3b0dced7d54ed23 WatchSource:0}: Error finding container b3b5b88f5ef860add6b6a00784b65374847adf5a2a7adf8aa3b0dced7d54ed23: Status 404 returned error can't find the container with id b3b5b88f5ef860add6b6a00784b65374847adf5a2a7adf8aa3b0dced7d54ed23 Sep 09 17:40:36.882064 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.882030 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p"] Sep 09 17:40:36.887640 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.887618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:36.890575 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.890548 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f636a1c6-dockercfg-lb8p2\"" Sep 09 17:40:36.890575 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.890556 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f636a1c6\"" Sep 09 17:40:36.895000 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:36.894974 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p"] Sep 09 17:40:37.015153 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.015120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a37c0ce-97ee-4220-8a32-bc9f129360fe-plugins-config-volume\") pod \"payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p\" (UID: \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\") " pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:37.015298 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.015268 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9n7m\" (UniqueName: \"kubernetes.io/projected/7a37c0ce-97ee-4220-8a32-bc9f129360fe-kube-api-access-g9n7m\") pod \"payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p\" (UID: \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\") " pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:37.116300 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.116266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9n7m\" (UniqueName: \"kubernetes.io/projected/7a37c0ce-97ee-4220-8a32-bc9f129360fe-kube-api-access-g9n7m\") pod \"payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p\" (UID: \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\") " pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:37.116490 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.116311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a37c0ce-97ee-4220-8a32-bc9f129360fe-plugins-config-volume\") pod \"payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p\" (UID: \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\") " pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:37.117018 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.116998 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a37c0ce-97ee-4220-8a32-bc9f129360fe-plugins-config-volume\") pod \"payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p\" (UID: \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\") " pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:37.123888 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.123868 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9n7m\" (UniqueName: \"kubernetes.io/projected/7a37c0ce-97ee-4220-8a32-bc9f129360fe-kube-api-access-g9n7m\") pod \"payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p\" (UID: \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\") " pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:37.203830 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.203796 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:37.422261 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.422239 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" event={"ID":"5c9501e0-9c45-4918-bd7e-d09927651402","Type":"ContainerStarted","Data":"b3b5b88f5ef860add6b6a00784b65374847adf5a2a7adf8aa3b0dced7d54ed23"} Sep 09 17:40:37.475546 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:37.475514 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p"] Sep 09 17:40:37.476767 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:40:37.476736 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a37c0ce_97ee_4220_8a32_bc9f129360fe.slice/crio-1aceb7ec9c04f957fea6ac29ecab59c78c4d1f6120292607c8eb6a553e3c25fa WatchSource:0}: Error finding container 1aceb7ec9c04f957fea6ac29ecab59c78c4d1f6120292607c8eb6a553e3c25fa: Status 404 returned error can't find the container with id 1aceb7ec9c04f957fea6ac29ecab59c78c4d1f6120292607c8eb6a553e3c25fa Sep 09 17:40:38.429905 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:38.429852 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" event={"ID":"5c9501e0-9c45-4918-bd7e-d09927651402","Type":"ContainerStarted","Data":"ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1"} Sep 09 17:40:38.430431 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:38.429961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:38.431938 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:38.431911 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" event={"ID":"7a37c0ce-97ee-4220-8a32-bc9f129360fe","Type":"ContainerStarted","Data":"771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8"} Sep 09 17:40:38.432072 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:38.431959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" event={"ID":"7a37c0ce-97ee-4220-8a32-bc9f129360fe","Type":"ContainerStarted","Data":"1aceb7ec9c04f957fea6ac29ecab59c78c4d1f6120292607c8eb6a553e3c25fa"} Sep 09 17:40:38.432124 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:38.432068 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:40:38.454329 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:38.454272 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" podStartSLOduration=1.8338115240000001 podStartE2EDuration="2.454257504s" totalImagesPullingTime="620.44598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:40:38.451959811 +0000 UTC m=+1432.800321287" watchObservedRunningTime="2026-09-09 17:40:38.454257504 +0000 UTC m=+1432.802618975" Sep 09 17:40:38.489509 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:38.489448 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" podStartSLOduration=2.489429983 podStartE2EDuration="2.489429983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:40:38.486142041 +0000 UTC m=+1432.834503513" watchObservedRunningTime="2026-09-09 17:40:38.489429983 +0000 UTC m=+1432.837791501" Sep 09 17:40:44.444129 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:44.444098 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:40:46.360381 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:46.360347 2580 scope.go:117] "RemoveContainer" containerID="ea4cfcd38e62c6844964549b9965ce753cdd4c94f5f0b9b611ad52d2a98f6d95" Sep 09 17:40:46.368912 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:46.368894 2580 scope.go:117] "RemoveContainer" containerID="619cbede42f3e7858040c982ad0dd0f37b89ade667c45b80b0e4aae9deb4b51c" Sep 09 17:40:48.697807 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:48.697775 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:40:49.439945 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:40:49.439914 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:41:07.778926 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:07.778890 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:41:19.186474 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:19.186435 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:41:37.184469 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:37.184427 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:41:39.810814 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:39.810780 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb"] Sep 09 17:41:39.811276 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:39.811054 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" podUID="5c9501e0-9c45-4918-bd7e-d09927651402" containerName="maas-api" containerID="cri-o://ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1" gracePeriod=30 Sep 09 17:41:39.876742 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:39.876669 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p"] Sep 09 17:41:39.877029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:39.876991 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" podUID="7a37c0ce-97ee-4220-8a32-bc9f129360fe" containerName="payload-processing" containerID="cri-o://771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8" gracePeriod=30 Sep 09 17:41:40.134412 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.134383 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:41:40.172307 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.172273 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:41:40.231325 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.231288 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-metrics-certs\") pod \"5c9501e0-9c45-4918-bd7e-d09927651402\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " Sep 09 17:41:40.231516 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.231334 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pgzp\" (UniqueName: \"kubernetes.io/projected/5c9501e0-9c45-4918-bd7e-d09927651402-kube-api-access-4pgzp\") pod \"5c9501e0-9c45-4918-bd7e-d09927651402\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " Sep 09 17:41:40.231516 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.231361 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-maas-api-tls\") pod \"5c9501e0-9c45-4918-bd7e-d09927651402\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " Sep 09 17:41:40.231516 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.231396 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9501e0-9c45-4918-bd7e-d09927651402-openshift-service-ca\") pod \"5c9501e0-9c45-4918-bd7e-d09927651402\" (UID: \"5c9501e0-9c45-4918-bd7e-d09927651402\") " Sep 09 17:41:40.231516 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.231411 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9n7m\" (UniqueName: \"kubernetes.io/projected/7a37c0ce-97ee-4220-8a32-bc9f129360fe-kube-api-access-g9n7m\") pod \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\" (UID: \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\") " Sep 09 17:41:40.231516 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.231462 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a37c0ce-97ee-4220-8a32-bc9f129360fe-plugins-config-volume\") pod \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\" (UID: \"7a37c0ce-97ee-4220-8a32-bc9f129360fe\") " Sep 09 17:41:40.232078 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.232048 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9501e0-9c45-4918-bd7e-d09927651402-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c9501e0-9c45-4918-bd7e-d09927651402" (UID: "5c9501e0-9c45-4918-bd7e-d09927651402"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:41:40.232172 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.232074 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a37c0ce-97ee-4220-8a32-bc9f129360fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a37c0ce-97ee-4220-8a32-bc9f129360fe" (UID: "7a37c0ce-97ee-4220-8a32-bc9f129360fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:41:40.233611 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.233586 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c9501e0-9c45-4918-bd7e-d09927651402" (UID: "5c9501e0-9c45-4918-bd7e-d09927651402"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:41:40.233700 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.233621 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c9501e0-9c45-4918-bd7e-d09927651402" (UID: "5c9501e0-9c45-4918-bd7e-d09927651402"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:41:40.233700 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.233650 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9501e0-9c45-4918-bd7e-d09927651402-kube-api-access-4pgzp" (OuterVolumeSpecName: "kube-api-access-4pgzp") pod "5c9501e0-9c45-4918-bd7e-d09927651402" (UID: "5c9501e0-9c45-4918-bd7e-d09927651402"). InnerVolumeSpecName "kube-api-access-4pgzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:41:40.234183 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.234165 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a37c0ce-97ee-4220-8a32-bc9f129360fe-kube-api-access-g9n7m" (OuterVolumeSpecName: "kube-api-access-g9n7m") pod "7a37c0ce-97ee-4220-8a32-bc9f129360fe" (UID: "7a37c0ce-97ee-4220-8a32-bc9f129360fe"). InnerVolumeSpecName "kube-api-access-g9n7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:41:40.332765 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.332725 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:41:40.332765 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.332757 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgzp\" (UniqueName: \"kubernetes.io/projected/5c9501e0-9c45-4918-bd7e-d09927651402-kube-api-access-4pgzp\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:41:40.332765 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.332768 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c9501e0-9c45-4918-bd7e-d09927651402-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:41:40.332765 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.332776 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c9501e0-9c45-4918-bd7e-d09927651402-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:41:40.333029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.332784 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9n7m\" (UniqueName: \"kubernetes.io/projected/7a37c0ce-97ee-4220-8a32-bc9f129360fe-kube-api-access-g9n7m\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:41:40.333029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.332793 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a37c0ce-97ee-4220-8a32-bc9f129360fe-plugins-config-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:41:40.794528 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.794492 2580 generic.go:356] "Generic (PLEG): container finished" podID="7a37c0ce-97ee-4220-8a32-bc9f129360fe" containerID="771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8" exitCode=0 Sep 09 17:41:40.794762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.794533 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" Sep 09 17:41:40.794762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.794573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" event={"ID":"7a37c0ce-97ee-4220-8a32-bc9f129360fe","Type":"ContainerDied","Data":"771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8"} Sep 09 17:41:40.794762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.794627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p" event={"ID":"7a37c0ce-97ee-4220-8a32-bc9f129360fe","Type":"ContainerDied","Data":"1aceb7ec9c04f957fea6ac29ecab59c78c4d1f6120292607c8eb6a553e3c25fa"} Sep 09 17:41:40.794762 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.794655 2580 scope.go:117] "RemoveContainer" containerID="771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8" Sep 09 17:41:40.796023 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.795995 2580 generic.go:356] "Generic (PLEG): container finished" podID="5c9501e0-9c45-4918-bd7e-d09927651402" containerID="ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1" exitCode=0 Sep 09 17:41:40.796150 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.796044 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" Sep 09 17:41:40.796211 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.796174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" event={"ID":"5c9501e0-9c45-4918-bd7e-d09927651402","Type":"ContainerDied","Data":"ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1"} Sep 09 17:41:40.796251 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.796216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb" event={"ID":"5c9501e0-9c45-4918-bd7e-d09927651402","Type":"ContainerDied","Data":"b3b5b88f5ef860add6b6a00784b65374847adf5a2a7adf8aa3b0dced7d54ed23"} Sep 09 17:41:40.805667 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.805627 2580 scope.go:117] "RemoveContainer" containerID="771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8" Sep 09 17:41:40.806115 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:41:40.806097 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8\": container with ID starting with 771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8 not found: ID does not exist" containerID="771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8" Sep 09 17:41:40.806184 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.806125 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8"} err="failed to get container status \"771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8\": rpc error: code = NotFound desc = could not find container \"771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8\": container with ID starting with 771178d9d4cea23564d885d9a6b47dce4f9fb3fb6abc151d00b3662eb8fd75d8 not found: ID does not exist" Sep 09 17:41:40.806184 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.806143 2580 scope.go:117] "RemoveContainer" containerID="ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1" Sep 09 17:41:40.816095 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.815548 2580 scope.go:117] "RemoveContainer" containerID="ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1" Sep 09 17:41:40.816095 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:41:40.815979 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1\": container with ID starting with ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1 not found: ID does not exist" containerID="ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1" Sep 09 17:41:40.816095 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.816039 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1"} err="failed to get container status \"ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1\": rpc error: code = NotFound desc = could not find container \"ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1\": container with ID starting with ed37467b1d0d7d84ca22029eccb6d94799a1864b5479805ee05a3ee09a7107f1 not found: ID does not exist" Sep 09 17:41:40.818308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.818286 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p"] Sep 09 17:41:40.824194 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.824166 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f636a1c6-9f856bcfc-nlr8p"] Sep 09 17:41:40.836172 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.836133 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb"] Sep 09 17:41:40.841217 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:40.841190 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f636a1c6-7dfd6bcd7d-mjddb"] Sep 09 17:41:42.132370 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:42.132325 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c9501e0-9c45-4918-bd7e-d09927651402" path="/var/lib/kubelet/pods/5c9501e0-9c45-4918-bd7e-d09927651402/volumes" Sep 09 17:41:42.132877 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:42.132850 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a37c0ce-97ee-4220-8a32-bc9f129360fe" path="/var/lib/kubelet/pods/7a37c0ce-97ee-4220-8a32-bc9f129360fe/volumes" Sep 09 17:41:46.290281 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:46.290250 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:41:46.297104 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:46.297083 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:41:46.297471 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:46.297455 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:41:46.300478 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:46.300454 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:41:48.680586 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:41:48.680548 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:42:40.781604 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:42:40.781520 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:42:46.432063 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:42:46.432016 2580 scope.go:117] "RemoveContainer" containerID="b04224dee7c1086cbc1c77026ca7cd9e64b19ce47503bc934d9b4dce4ca1819d" Sep 09 17:42:50.088639 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:42:50.088591 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:43:35.094566 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.094525 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz"] Sep 09 17:43:35.095225 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.095203 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c9501e0-9c45-4918-bd7e-d09927651402" containerName="maas-api" Sep 09 17:43:35.095308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.095227 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c9501e0-9c45-4918-bd7e-d09927651402" containerName="maas-api" Sep 09 17:43:35.095308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.095278 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a37c0ce-97ee-4220-8a32-bc9f129360fe" containerName="payload-processing" Sep 09 17:43:35.095308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.095290 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a37c0ce-97ee-4220-8a32-bc9f129360fe" containerName="payload-processing" Sep 09 17:43:35.095456 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.095411 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c9501e0-9c45-4918-bd7e-d09927651402" containerName="maas-api" Sep 09 17:43:35.095456 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.095425 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a37c0ce-97ee-4220-8a32-bc9f129360fe" containerName="payload-processing" Sep 09 17:43:35.098784 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.098761 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.101551 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.101528 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cf91eb-serving-cert\"" Sep 09 17:43:35.109041 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.109008 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz"] Sep 09 17:43:35.142018 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.141985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4403c382-221e-4a75-98fb-58c5d5e3089c-openshift-service-ca\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.142199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.142067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.142274 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.142217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-metrics-certs\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.142274 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.142264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wwrk\" (UniqueName: \"kubernetes.io/projected/4403c382-221e-4a75-98fb-58c5d5e3089c-kube-api-access-7wwrk\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.243788 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.243746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4403c382-221e-4a75-98fb-58c5d5e3089c-openshift-service-ca\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.243997 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.243829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.243997 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.243893 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-metrics-certs\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.243997 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.243915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wwrk\" (UniqueName: \"kubernetes.io/projected/4403c382-221e-4a75-98fb-58c5d5e3089c-kube-api-access-7wwrk\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.243997 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:43:35.243962 2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-serving-cert: secret "maas-api-e2e-api-a-cf91eb-serving-cert" not found Sep 09 17:43:35.244215 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:43:35.244022 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls podName:4403c382-221e-4a75-98fb-58c5d5e3089c nodeName:}" failed. No retries permitted until 2026-09-09 17:43:35.744006581 +0000 UTC m=+1610.092368038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls") pod "maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" (UID: "4403c382-221e-4a75-98fb-58c5d5e3089c") : secret "maas-api-e2e-api-a-cf91eb-serving-cert" not found Sep 09 17:43:35.244432 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.244402 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4403c382-221e-4a75-98fb-58c5d5e3089c-openshift-service-ca\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.246333 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.246311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-metrics-certs\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.251983 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.251957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wwrk\" (UniqueName: \"kubernetes.io/projected/4403c382-221e-4a75-98fb-58c5d5e3089c-kube-api-access-7wwrk\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.745575 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.745538 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w"] Sep 09 17:43:35.748612 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.748590 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.749383 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.749359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:35.751154 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.751135 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls\") pod \"maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:35.753773 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.753755 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cf91eb-dockercfg-rhkt7\"" Sep 09 17:43:35.753876 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.753774 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cf91eb\"" Sep 09 17:43:35.760980 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.760953 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w"] Sep 09 17:43:35.849381 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.849346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxq2\" (UniqueName: \"kubernetes.io/projected/bc088ed2-9869-4f39-aee7-abfab6bded71-kube-api-access-jjxq2\") pod \"payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w\" (UID: \"bc088ed2-9869-4f39-aee7-abfab6bded71\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:35.849550 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.849395 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc088ed2-9869-4f39-aee7-abfab6bded71-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w\" (UID: \"bc088ed2-9869-4f39-aee7-abfab6bded71\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:35.950838 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.950803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxq2\" (UniqueName: \"kubernetes.io/projected/bc088ed2-9869-4f39-aee7-abfab6bded71-kube-api-access-jjxq2\") pod \"payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w\" (UID: \"bc088ed2-9869-4f39-aee7-abfab6bded71\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:35.951017 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.950853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc088ed2-9869-4f39-aee7-abfab6bded71-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w\" (UID: \"bc088ed2-9869-4f39-aee7-abfab6bded71\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:35.951539 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.951516 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc088ed2-9869-4f39-aee7-abfab6bded71-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w\" (UID: \"bc088ed2-9869-4f39-aee7-abfab6bded71\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:35.958621 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:35.958601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxq2\" (UniqueName: \"kubernetes.io/projected/bc088ed2-9869-4f39-aee7-abfab6bded71-kube-api-access-jjxq2\") pod \"payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w\" (UID: \"bc088ed2-9869-4f39-aee7-abfab6bded71\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:36.010645 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.010622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:36.071589 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.071544 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:36.284498 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.284466 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz"] Sep 09 17:43:36.286653 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:43:36.286631 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4403c382_221e_4a75_98fb_58c5d5e3089c.slice/crio-eb2fd996e8ae49a36be6aa3d12485c59ae7f5328653cd309f44d6cc89b6ce3fb WatchSource:0}: Error finding container eb2fd996e8ae49a36be6aa3d12485c59ae7f5328653cd309f44d6cc89b6ce3fb: Status 404 returned error can't find the container with id eb2fd996e8ae49a36be6aa3d12485c59ae7f5328653cd309f44d6cc89b6ce3fb Sep 09 17:43:36.354222 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.354196 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w"] Sep 09 17:43:36.355678 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:43:36.355656 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc088ed2_9869_4f39_aee7_abfab6bded71.slice/crio-1f302a2a5ff3fc5b58d167f26b4084c80665d667aad20a7b728c97c75dedf08d WatchSource:0}: Error finding container 1f302a2a5ff3fc5b58d167f26b4084c80665d667aad20a7b728c97c75dedf08d: Status 404 returned error can't find the container with id 1f302a2a5ff3fc5b58d167f26b4084c80665d667aad20a7b728c97c75dedf08d Sep 09 17:43:36.466776 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.466739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" event={"ID":"bc088ed2-9869-4f39-aee7-abfab6bded71","Type":"ContainerStarted","Data":"a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1"} Sep 09 17:43:36.466962 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.466799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" event={"ID":"bc088ed2-9869-4f39-aee7-abfab6bded71","Type":"ContainerStarted","Data":"1f302a2a5ff3fc5b58d167f26b4084c80665d667aad20a7b728c97c75dedf08d"} Sep 09 17:43:36.466962 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.466928 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:43:36.468084 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.468060 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" event={"ID":"4403c382-221e-4a75-98fb-58c5d5e3089c","Type":"ContainerStarted","Data":"eb2fd996e8ae49a36be6aa3d12485c59ae7f5328653cd309f44d6cc89b6ce3fb"} Sep 09 17:43:36.484136 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:36.484092 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" podStartSLOduration=1.484077207 podStartE2EDuration="1.484077207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:43:36.482840882 +0000 UTC m=+1610.831202394" watchObservedRunningTime="2026-09-09 17:43:36.484077207 +0000 UTC m=+1610.832438679" Sep 09 17:43:37.475133 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:37.475090 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" event={"ID":"4403c382-221e-4a75-98fb-58c5d5e3089c","Type":"ContainerStarted","Data":"0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68"} Sep 09 17:43:37.475537 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:37.475380 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:37.492670 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:37.492623 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" podStartSLOduration=1.875889963 podStartE2EDuration="2.492609906s" totalImagesPullingTime="616.719943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:43:37.490940882 +0000 UTC m=+1611.839302369" watchObservedRunningTime="2026-09-09 17:43:37.492609906 +0000 UTC m=+1611.840971369" Sep 09 17:43:39.784443 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:39.784359 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:43:43.486347 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:43.486317 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:43:47.477394 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:43:47.477363 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:44:04.171622 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.171591 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql"] Sep 09 17:44:04.175813 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.175787 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.178217 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.178193 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ecf58e-serving-cert\"" Sep 09 17:44:04.188598 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.188564 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql"] Sep 09 17:44:04.330663 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.330626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b00a756-6fc1-4043-bdd8-45599866e7a4-openshift-service-ca\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.330663 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.330663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-metrics-certs\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.330938 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.330794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxgbf\" (UniqueName: \"kubernetes.io/projected/2b00a756-6fc1-4043-bdd8-45599866e7a4-kube-api-access-wxgbf\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.330980 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.330933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-maas-api-tls\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.432084 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.431986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxgbf\" (UniqueName: \"kubernetes.io/projected/2b00a756-6fc1-4043-bdd8-45599866e7a4-kube-api-access-wxgbf\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.432287 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.432094 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-maas-api-tls\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.432287 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.432122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b00a756-6fc1-4043-bdd8-45599866e7a4-openshift-service-ca\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.432287 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.432144 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-metrics-certs\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.432957 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.432928 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b00a756-6fc1-4043-bdd8-45599866e7a4-openshift-service-ca\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.434605 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.434585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-maas-api-tls\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.434704 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.434632 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-metrics-certs\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.440477 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.440439 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxgbf\" (UniqueName: \"kubernetes.io/projected/2b00a756-6fc1-4043-bdd8-45599866e7a4-kube-api-access-wxgbf\") pod \"maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.490417 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.490387 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:04.765316 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:04.765285 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql"] Sep 09 17:44:04.768782 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:44:04.768753 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b00a756_6fc1_4043_bdd8_45599866e7a4.slice/crio-1f9ecfba8548f70cb21cda9e78bd52f15b7806c619b8de2c33c9fc43992cffa7 WatchSource:0}: Error finding container 1f9ecfba8548f70cb21cda9e78bd52f15b7806c619b8de2c33c9fc43992cffa7: Status 404 returned error can't find the container with id 1f9ecfba8548f70cb21cda9e78bd52f15b7806c619b8de2c33c9fc43992cffa7 Sep 09 17:44:05.638601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:05.638515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" event={"ID":"2b00a756-6fc1-4043-bdd8-45599866e7a4","Type":"ContainerStarted","Data":"13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1"} Sep 09 17:44:05.638601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:05.638579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" event={"ID":"2b00a756-6fc1-4043-bdd8-45599866e7a4","Type":"ContainerStarted","Data":"1f9ecfba8548f70cb21cda9e78bd52f15b7806c619b8de2c33c9fc43992cffa7"} Sep 09 17:44:05.639043 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:05.638748 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:05.657934 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:05.657892 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" podStartSLOduration=1.10564487 podStartE2EDuration="1.65787733s" totalImagesPullingTime="552.23246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:44:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:44:05.657858377 +0000 UTC m=+1640.006219866" watchObservedRunningTime="2026-09-09 17:44:05.65787733 +0000 UTC m=+1640.006238793" Sep 09 17:44:09.781412 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:09.781374 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:44:11.649301 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:11.649271 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:44:46.484883 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.484847 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz"] Sep 09 17:44:46.485466 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.485155 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" podUID="4403c382-221e-4a75-98fb-58c5d5e3089c" containerName="maas-api" containerID="cri-o://0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68" gracePeriod=30 Sep 09 17:44:46.535007 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.534826 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w"] Sep 09 17:44:46.535771 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.535420 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" podUID="bc088ed2-9869-4f39-aee7-abfab6bded71" containerName="payload-processing" containerID="cri-o://a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1" gracePeriod=30 Sep 09 17:44:46.783180 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.783152 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:44:46.819162 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.819033 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:44:46.833206 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833139 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4403c382-221e-4a75-98fb-58c5d5e3089c-openshift-service-ca\") pod \"4403c382-221e-4a75-98fb-58c5d5e3089c\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " Sep 09 17:44:46.833206 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833179 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc088ed2-9869-4f39-aee7-abfab6bded71-plugins-config-volume\") pod \"bc088ed2-9869-4f39-aee7-abfab6bded71\" (UID: \"bc088ed2-9869-4f39-aee7-abfab6bded71\") " Sep 09 17:44:46.833453 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833285 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls\") pod \"4403c382-221e-4a75-98fb-58c5d5e3089c\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " Sep 09 17:44:46.833453 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833360 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-metrics-certs\") pod \"4403c382-221e-4a75-98fb-58c5d5e3089c\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " Sep 09 17:44:46.833453 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833431 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjxq2\" (UniqueName: \"kubernetes.io/projected/bc088ed2-9869-4f39-aee7-abfab6bded71-kube-api-access-jjxq2\") pod \"bc088ed2-9869-4f39-aee7-abfab6bded71\" (UID: \"bc088ed2-9869-4f39-aee7-abfab6bded71\") " Sep 09 17:44:46.833616 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833463 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wwrk\" (UniqueName: \"kubernetes.io/projected/4403c382-221e-4a75-98fb-58c5d5e3089c-kube-api-access-7wwrk\") pod \"4403c382-221e-4a75-98fb-58c5d5e3089c\" (UID: \"4403c382-221e-4a75-98fb-58c5d5e3089c\") " Sep 09 17:44:46.833616 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833521 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4403c382-221e-4a75-98fb-58c5d5e3089c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4403c382-221e-4a75-98fb-58c5d5e3089c" (UID: "4403c382-221e-4a75-98fb-58c5d5e3089c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:44:46.833616 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833533 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc088ed2-9869-4f39-aee7-abfab6bded71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc088ed2-9869-4f39-aee7-abfab6bded71" (UID: "bc088ed2-9869-4f39-aee7-abfab6bded71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:44:46.833910 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833891 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4403c382-221e-4a75-98fb-58c5d5e3089c-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:44:46.833984 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.833916 2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc088ed2-9869-4f39-aee7-abfab6bded71-plugins-config-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:44:46.836076 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.835983 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4403c382-221e-4a75-98fb-58c5d5e3089c" (UID: "4403c382-221e-4a75-98fb-58c5d5e3089c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:44:46.837155 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.837124 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc088ed2-9869-4f39-aee7-abfab6bded71-kube-api-access-jjxq2" (OuterVolumeSpecName: "kube-api-access-jjxq2") pod "bc088ed2-9869-4f39-aee7-abfab6bded71" (UID: "bc088ed2-9869-4f39-aee7-abfab6bded71"). InnerVolumeSpecName "kube-api-access-jjxq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:44:46.837687 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.837659 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4403c382-221e-4a75-98fb-58c5d5e3089c" (UID: "4403c382-221e-4a75-98fb-58c5d5e3089c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:44:46.838499 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.838475 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4403c382-221e-4a75-98fb-58c5d5e3089c-kube-api-access-7wwrk" (OuterVolumeSpecName: "kube-api-access-7wwrk") pod "4403c382-221e-4a75-98fb-58c5d5e3089c" (UID: "4403c382-221e-4a75-98fb-58c5d5e3089c"). InnerVolumeSpecName "kube-api-access-7wwrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:44:46.869995 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.869959 2580 generic.go:356] "Generic (PLEG): container finished" podID="bc088ed2-9869-4f39-aee7-abfab6bded71" containerID="a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1" exitCode=0 Sep 09 17:44:46.870147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.869998 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" Sep 09 17:44:46.870147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.870043 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" event={"ID":"bc088ed2-9869-4f39-aee7-abfab6bded71","Type":"ContainerDied","Data":"a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1"} Sep 09 17:44:46.870147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.870117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w" event={"ID":"bc088ed2-9869-4f39-aee7-abfab6bded71","Type":"ContainerDied","Data":"1f302a2a5ff3fc5b58d167f26b4084c80665d667aad20a7b728c97c75dedf08d"} Sep 09 17:44:46.870147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.870139 2580 scope.go:117] "RemoveContainer" containerID="a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1" Sep 09 17:44:46.871440 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.871419 2580 generic.go:356] "Generic (PLEG): container finished" podID="4403c382-221e-4a75-98fb-58c5d5e3089c" containerID="0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68" exitCode=0 Sep 09 17:44:46.871562 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.871456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" event={"ID":"4403c382-221e-4a75-98fb-58c5d5e3089c","Type":"ContainerDied","Data":"0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68"} Sep 09 17:44:46.871562 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.871482 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" Sep 09 17:44:46.871562 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.871491 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz" event={"ID":"4403c382-221e-4a75-98fb-58c5d5e3089c","Type":"ContainerDied","Data":"eb2fd996e8ae49a36be6aa3d12485c59ae7f5328653cd309f44d6cc89b6ce3fb"} Sep 09 17:44:46.879178 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.879162 2580 scope.go:117] "RemoveContainer" containerID="a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1" Sep 09 17:44:46.879461 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:44:46.879438 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1\": container with ID starting with a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1 not found: ID does not exist" containerID="a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1" Sep 09 17:44:46.879555 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.879467 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1"} err="failed to get container status \"a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1\": rpc error: code = NotFound desc = could not find container \"a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1\": container with ID starting with a3689aef1aae93da6c7b23b1a960b10ed85dd004466078a8e8e2282f60d19de1 not found: ID does not exist" Sep 09 17:44:46.879555 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.879482 2580 scope.go:117] "RemoveContainer" containerID="0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68" Sep 09 17:44:46.888775 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.888757 2580 scope.go:117] "RemoveContainer" containerID="0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68" Sep 09 17:44:46.889025 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:44:46.889007 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68\": container with ID starting with 0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68 not found: ID does not exist" containerID="0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68" Sep 09 17:44:46.889078 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.889029 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68"} err="failed to get container status \"0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68\": rpc error: code = NotFound desc = could not find container \"0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68\": container with ID starting with 0c996d636aacc4791db2fffb8895a57e9445b772d751f53dde017376847faa68 not found: ID does not exist" Sep 09 17:44:46.896941 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.896922 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz"] Sep 09 17:44:46.900341 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.900318 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cf91eb-778c7dff59-6drjz"] Sep 09 17:44:46.911503 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.911474 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w"] Sep 09 17:44:46.921585 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.921561 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf91eb-579cb94979-4qd4w"] Sep 09 17:44:46.935131 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.935101 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:44:46.935131 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.935131 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4403c382-221e-4a75-98fb-58c5d5e3089c-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:44:46.935279 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.935145 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjxq2\" (UniqueName: \"kubernetes.io/projected/bc088ed2-9869-4f39-aee7-abfab6bded71-kube-api-access-jjxq2\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:44:46.935279 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:46.935159 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wwrk\" (UniqueName: \"kubernetes.io/projected/4403c382-221e-4a75-98fb-58c5d5e3089c-kube-api-access-7wwrk\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:44:48.132051 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:48.132010 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4403c382-221e-4a75-98fb-58c5d5e3089c" path="/var/lib/kubelet/pods/4403c382-221e-4a75-98fb-58c5d5e3089c/volumes" Sep 09 17:44:48.132560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:48.132542 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc088ed2-9869-4f39-aee7-abfab6bded71" path="/var/lib/kubelet/pods/bc088ed2-9869-4f39-aee7-abfab6bded71/volumes" Sep 09 17:44:49.886132 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:44:49.886091 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:45:08.984269 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:08.984177 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql"] Sep 09 17:45:08.984753 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:08.984433 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" podUID="2b00a756-6fc1-4043-bdd8-45599866e7a4" containerName="maas-api" containerID="cri-o://13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1" gracePeriod=30 Sep 09 17:45:09.234242 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.234212 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:45:09.357115 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.357080 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b00a756-6fc1-4043-bdd8-45599866e7a4-openshift-service-ca\") pod \"2b00a756-6fc1-4043-bdd8-45599866e7a4\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " Sep 09 17:45:09.357325 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.357121 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-metrics-certs\") pod \"2b00a756-6fc1-4043-bdd8-45599866e7a4\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " Sep 09 17:45:09.357325 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.357139 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-maas-api-tls\") pod \"2b00a756-6fc1-4043-bdd8-45599866e7a4\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " Sep 09 17:45:09.357325 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.357228 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxgbf\" (UniqueName: \"kubernetes.io/projected/2b00a756-6fc1-4043-bdd8-45599866e7a4-kube-api-access-wxgbf\") pod \"2b00a756-6fc1-4043-bdd8-45599866e7a4\" (UID: \"2b00a756-6fc1-4043-bdd8-45599866e7a4\") " Sep 09 17:45:09.357557 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.357527 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b00a756-6fc1-4043-bdd8-45599866e7a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b00a756-6fc1-4043-bdd8-45599866e7a4" (UID: "2b00a756-6fc1-4043-bdd8-45599866e7a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:45:09.360077 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.360040 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b00a756-6fc1-4043-bdd8-45599866e7a4" (UID: "2b00a756-6fc1-4043-bdd8-45599866e7a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:45:09.360200 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.360179 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b00a756-6fc1-4043-bdd8-45599866e7a4-kube-api-access-wxgbf" (OuterVolumeSpecName: "kube-api-access-wxgbf") pod "2b00a756-6fc1-4043-bdd8-45599866e7a4" (UID: "2b00a756-6fc1-4043-bdd8-45599866e7a4"). InnerVolumeSpecName "kube-api-access-wxgbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:45:09.360407 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.360367 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b00a756-6fc1-4043-bdd8-45599866e7a4" (UID: "2b00a756-6fc1-4043-bdd8-45599866e7a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:45:09.458843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.458800 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b00a756-6fc1-4043-bdd8-45599866e7a4-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:09.458843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.458834 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:09.458843 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.458846 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b00a756-6fc1-4043-bdd8-45599866e7a4-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:09.459088 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:09.458856 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxgbf\" (UniqueName: \"kubernetes.io/projected/2b00a756-6fc1-4043-bdd8-45599866e7a4-kube-api-access-wxgbf\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:10.014288 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.014256 2580 generic.go:356] "Generic (PLEG): container finished" podID="2b00a756-6fc1-4043-bdd8-45599866e7a4" containerID="13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1" exitCode=0 Sep 09 17:45:10.014749 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.014306 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" Sep 09 17:45:10.014749 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.014309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" event={"ID":"2b00a756-6fc1-4043-bdd8-45599866e7a4","Type":"ContainerDied","Data":"13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1"} Sep 09 17:45:10.014749 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.014366 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql" event={"ID":"2b00a756-6fc1-4043-bdd8-45599866e7a4","Type":"ContainerDied","Data":"1f9ecfba8548f70cb21cda9e78bd52f15b7806c619b8de2c33c9fc43992cffa7"} Sep 09 17:45:10.014749 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.014393 2580 scope.go:117] "RemoveContainer" containerID="13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1" Sep 09 17:45:10.024041 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.024017 2580 scope.go:117] "RemoveContainer" containerID="13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1" Sep 09 17:45:10.024317 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:45:10.024297 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1\": container with ID starting with 13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1 not found: ID does not exist" containerID="13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1" Sep 09 17:45:10.024364 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.024324 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1"} err="failed to get container status \"13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1\": rpc error: code = NotFound desc = could not find container \"13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1\": container with ID starting with 13fd2a12f98e6c4127caf88d6de5d693ae5e8f8c2c4259e55daa77390b7331c1 not found: ID does not exist" Sep 09 17:45:10.036807 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.036773 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql"] Sep 09 17:45:10.039021 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.038997 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ecf58e-7c466cb45b-74gql"] Sep 09 17:45:10.138271 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:10.138234 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b00a756-6fc1-4043-bdd8-45599866e7a4" path="/var/lib/kubelet/pods/2b00a756-6fc1-4043-bdd8-45599866e7a4/volumes" Sep 09 17:45:12.280309 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:12.280276 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:45:26.288419 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.288383 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c"] Sep 09 17:45:26.289027 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.288619 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" podUID="0aa76aa3-9f57-4b09-80c2-23c3185327b8" containerName="maas-api" containerID="cri-o://00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7" gracePeriod=30 Sep 09 17:45:26.535724 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.535689 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:45:26.720264 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.720175 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls\") pod \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " Sep 09 17:45:26.720264 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.720209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-metrics-certs\") pod \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " Sep 09 17:45:26.720264 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.720261 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa76aa3-9f57-4b09-80c2-23c3185327b8-openshift-service-ca\") pod \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " Sep 09 17:45:26.720534 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.720289 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmq5q\" (UniqueName: \"kubernetes.io/projected/0aa76aa3-9f57-4b09-80c2-23c3185327b8-kube-api-access-qmq5q\") pod \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\" (UID: \"0aa76aa3-9f57-4b09-80c2-23c3185327b8\") " Sep 09 17:45:26.720660 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.720625 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa76aa3-9f57-4b09-80c2-23c3185327b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0aa76aa3-9f57-4b09-80c2-23c3185327b8" (UID: "0aa76aa3-9f57-4b09-80c2-23c3185327b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:45:26.722453 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.722428 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa76aa3-9f57-4b09-80c2-23c3185327b8-kube-api-access-qmq5q" (OuterVolumeSpecName: "kube-api-access-qmq5q") pod "0aa76aa3-9f57-4b09-80c2-23c3185327b8" (UID: "0aa76aa3-9f57-4b09-80c2-23c3185327b8"). InnerVolumeSpecName "kube-api-access-qmq5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:45:26.722553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.722435 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0aa76aa3-9f57-4b09-80c2-23c3185327b8" (UID: "0aa76aa3-9f57-4b09-80c2-23c3185327b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:45:26.722553 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.722458 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0aa76aa3-9f57-4b09-80c2-23c3185327b8" (UID: "0aa76aa3-9f57-4b09-80c2-23c3185327b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:45:26.821255 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.821219 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:26.821255 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.821247 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0aa76aa3-9f57-4b09-80c2-23c3185327b8-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:26.821255 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.821259 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0aa76aa3-9f57-4b09-80c2-23c3185327b8-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:26.821490 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:26.821268 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmq5q\" (UniqueName: \"kubernetes.io/projected/0aa76aa3-9f57-4b09-80c2-23c3185327b8-kube-api-access-qmq5q\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:27.110289 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.110251 2580 generic.go:356] "Generic (PLEG): container finished" podID="0aa76aa3-9f57-4b09-80c2-23c3185327b8" containerID="00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7" exitCode=0 Sep 09 17:45:27.110488 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.110299 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" Sep 09 17:45:27.110488 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.110336 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" event={"ID":"0aa76aa3-9f57-4b09-80c2-23c3185327b8","Type":"ContainerDied","Data":"00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7"} Sep 09 17:45:27.110488 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.110386 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c" event={"ID":"0aa76aa3-9f57-4b09-80c2-23c3185327b8","Type":"ContainerDied","Data":"a28ffd52a88afd90bf8504df324f5cae8ed10661044aa2b75a3fae276b31b280"} Sep 09 17:45:27.110488 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.110410 2580 scope.go:117] "RemoveContainer" containerID="00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7" Sep 09 17:45:27.120479 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.120461 2580 scope.go:117] "RemoveContainer" containerID="00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7" Sep 09 17:45:27.120771 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:45:27.120749 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7\": container with ID starting with 00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7 not found: ID does not exist" containerID="00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7" Sep 09 17:45:27.120882 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.120778 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7"} err="failed to get container status \"00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7\": rpc error: code = NotFound desc = could not find container \"00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7\": container with ID starting with 00fe05bd42eaa9f207313672401cb36a75c9ec4e53207e250c661b2e366a3da7 not found: ID does not exist" Sep 09 17:45:27.133604 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.133576 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c"] Sep 09 17:45:27.137641 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:27.137620 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6b44e3-8686569ddb-s7j8c"] Sep 09 17:45:28.131985 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:28.131943 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aa76aa3-9f57-4b09-80c2-23c3185327b8" path="/var/lib/kubelet/pods/0aa76aa3-9f57-4b09-80c2-23c3185327b8/volumes" Sep 09 17:45:29.379421 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:29.379384 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:45:43.927644 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:43.927608 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc"] Sep 09 17:45:43.928184 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:43.927869 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" podUID="e68411a2-7758-427b-973c-d0b04f3175b3" containerName="maas-api" containerID="cri-o://075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604" gracePeriod=30 Sep 09 17:45:44.186369 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.186313 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:45:44.222324 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.222284 2580 generic.go:356] "Generic (PLEG): container finished" podID="e68411a2-7758-427b-973c-d0b04f3175b3" containerID="075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604" exitCode=0 Sep 09 17:45:44.222500 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.222358 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" Sep 09 17:45:44.222500 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.222361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" event={"ID":"e68411a2-7758-427b-973c-d0b04f3175b3","Type":"ContainerDied","Data":"075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604"} Sep 09 17:45:44.222619 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.222520 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc" event={"ID":"e68411a2-7758-427b-973c-d0b04f3175b3","Type":"ContainerDied","Data":"c9212d2e9b8b6e907d8ae62f42349a46694c58472c9965f561119c8e855c8ec0"} Sep 09 17:45:44.222619 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.222556 2580 scope.go:117] "RemoveContainer" containerID="075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604" Sep 09 17:45:44.233733 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.233696 2580 scope.go:117] "RemoveContainer" containerID="075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604" Sep 09 17:45:44.234234 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:45:44.234210 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604\": container with ID starting with 075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604 not found: ID does not exist" containerID="075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604" Sep 09 17:45:44.234328 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.234241 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604"} err="failed to get container status \"075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604\": rpc error: code = NotFound desc = could not find container \"075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604\": container with ID starting with 075040fe8907e862c31b7594e4f57d11a9df353c6a5a8e2997d51297ea2f1604 not found: ID does not exist" Sep 09 17:45:44.280422 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.280386 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-metrics-certs\") pod \"e68411a2-7758-427b-973c-d0b04f3175b3\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " Sep 09 17:45:44.280588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.280456 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68411a2-7758-427b-973c-d0b04f3175b3-openshift-service-ca\") pod \"e68411a2-7758-427b-973c-d0b04f3175b3\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " Sep 09 17:45:44.280588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.280507 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cznh\" (UniqueName: \"kubernetes.io/projected/e68411a2-7758-427b-973c-d0b04f3175b3-kube-api-access-2cznh\") pod \"e68411a2-7758-427b-973c-d0b04f3175b3\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " Sep 09 17:45:44.280588 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.280538 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls\") pod \"e68411a2-7758-427b-973c-d0b04f3175b3\" (UID: \"e68411a2-7758-427b-973c-d0b04f3175b3\") " Sep 09 17:45:44.280862 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.280835 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68411a2-7758-427b-973c-d0b04f3175b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e68411a2-7758-427b-973c-d0b04f3175b3" (UID: "e68411a2-7758-427b-973c-d0b04f3175b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:45:44.282558 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.282527 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e68411a2-7758-427b-973c-d0b04f3175b3" (UID: "e68411a2-7758-427b-973c-d0b04f3175b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:45:44.282662 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.282586 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e68411a2-7758-427b-973c-d0b04f3175b3" (UID: "e68411a2-7758-427b-973c-d0b04f3175b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:45:44.282733 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.282688 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68411a2-7758-427b-973c-d0b04f3175b3-kube-api-access-2cznh" (OuterVolumeSpecName: "kube-api-access-2cznh") pod "e68411a2-7758-427b-973c-d0b04f3175b3" (UID: "e68411a2-7758-427b-973c-d0b04f3175b3"). InnerVolumeSpecName "kube-api-access-2cznh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:45:44.382143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.382107 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68411a2-7758-427b-973c-d0b04f3175b3-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:44.382143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.382141 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cznh\" (UniqueName: \"kubernetes.io/projected/e68411a2-7758-427b-973c-d0b04f3175b3-kube-api-access-2cznh\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:44.382353 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.382153 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:44.382353 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.382165 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e68411a2-7758-427b-973c-d0b04f3175b3-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:45:44.545752 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.545699 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc"] Sep 09 17:45:44.547528 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:44.547506 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-607751-6cd5cf98c5-cd2tc"] Sep 09 17:45:46.132652 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:46.132615 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68411a2-7758-427b-973c-d0b04f3175b3" path="/var/lib/kubelet/pods/e68411a2-7758-427b-973c-d0b04f3175b3/volumes" Sep 09 17:45:46.580163 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:46.580129 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:45:55.090691 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:45:55.090657 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:46:46.336961 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:46:46.336928 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:46:46.340140 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:46:46.340115 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:46:46.343279 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:46:46.343258 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:46:46.346443 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:46:46.346425 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:46:46.498738 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:46:46.498684 2580 scope.go:117] "RemoveContainer" containerID="bf2127ff2f11cd323fafc58d13819c1872aa1a46a638000878a4e80ce1cd7d38" Sep 09 17:46:46.507281 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:46:46.507256 2580 scope.go:117] "RemoveContainer" containerID="17092a91b4af43985a0d4e4f436191975c6a4980ab2dfebd0580c97f8cd99b26" Sep 09 17:47:16.785591 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:47:16.785557 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:47:42.690986 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:47:42.690945 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:47:50.592485 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:47:50.592449 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:47:58.783287 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:47:58.783254 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:48:16.491023 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:16.490936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:48:22.786563 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:22.786520 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:48:25.909143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:25.909106 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf"] Sep 09 17:48:26.113495 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:26.113430 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" podUID="43f02d6c-644b-4910-a844-022f00f5e841" containerName="manager" containerID="cri-o://2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87" gracePeriod=10 Sep 09 17:48:26.953828 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:26.953802 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:48:26.989260 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:26.989226 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43f02d6c-644b-4910-a844-022f00f5e841-extensions-socket-volume\") pod \"43f02d6c-644b-4910-a844-022f00f5e841\" (UID: \"43f02d6c-644b-4910-a844-022f00f5e841\") " Sep 09 17:48:26.989441 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:26.989320 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnffc\" (UniqueName: \"kubernetes.io/projected/43f02d6c-644b-4910-a844-022f00f5e841-kube-api-access-xnffc\") pod \"43f02d6c-644b-4910-a844-022f00f5e841\" (UID: \"43f02d6c-644b-4910-a844-022f00f5e841\") " Sep 09 17:48:26.989601 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:26.989580 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f02d6c-644b-4910-a844-022f00f5e841-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "43f02d6c-644b-4910-a844-022f00f5e841" (UID: "43f02d6c-644b-4910-a844-022f00f5e841"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 17:48:26.991420 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:26.991388 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f02d6c-644b-4910-a844-022f00f5e841-kube-api-access-xnffc" (OuterVolumeSpecName: "kube-api-access-xnffc") pod "43f02d6c-644b-4910-a844-022f00f5e841" (UID: "43f02d6c-644b-4910-a844-022f00f5e841"). InnerVolumeSpecName "kube-api-access-xnffc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:48:27.090290 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.090206 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnffc\" (UniqueName: \"kubernetes.io/projected/43f02d6c-644b-4910-a844-022f00f5e841-kube-api-access-xnffc\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:48:27.090290 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.090233 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/43f02d6c-644b-4910-a844-022f00f5e841-extensions-socket-volume\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:48:27.120131 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.120099 2580 generic.go:356] "Generic (PLEG): container finished" podID="43f02d6c-644b-4910-a844-022f00f5e841" containerID="2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87" exitCode=0 Sep 09 17:48:27.120292 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.120151 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" Sep 09 17:48:27.120292 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.120188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" event={"ID":"43f02d6c-644b-4910-a844-022f00f5e841","Type":"ContainerDied","Data":"2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87"} Sep 09 17:48:27.120292 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.120245 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf" event={"ID":"43f02d6c-644b-4910-a844-022f00f5e841","Type":"ContainerDied","Data":"7dfc10644035aa8e1b17a4ebba1dd46620eed58a3ade83cb8d851440ce483236"} Sep 09 17:48:27.120292 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.120269 2580 scope.go:117] "RemoveContainer" containerID="2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87" Sep 09 17:48:27.130308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.130293 2580 scope.go:117] "RemoveContainer" containerID="2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87" Sep 09 17:48:27.130573 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:48:27.130553 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87\": container with ID starting with 2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87 not found: ID does not exist" containerID="2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87" Sep 09 17:48:27.130614 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.130582 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87"} err="failed to get container status \"2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87\": rpc error: code = NotFound desc = could not find container \"2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87\": container with ID starting with 2fef875768291a35e85b5c8746673625c449815c754c7732b82729ab09233f87 not found: ID does not exist" Sep 09 17:48:27.141136 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.141116 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf"] Sep 09 17:48:27.144480 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:27.144459 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2blhf"] Sep 09 17:48:28.131073 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:48:28.131039 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f02d6c-644b-4910-a844-022f00f5e841" path="/var/lib/kubelet/pods/43f02d6c-644b-4910-a844-022f00f5e841/volumes" Sep 09 17:49:26.608194 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608160 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw"] Sep 09 17:49:26.608602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608552 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f02d6c-644b-4910-a844-022f00f5e841" containerName="manager" Sep 09 17:49:26.608602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608563 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f02d6c-644b-4910-a844-022f00f5e841" containerName="manager" Sep 09 17:49:26.608602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608571 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc088ed2-9869-4f39-aee7-abfab6bded71" containerName="payload-processing" Sep 09 17:49:26.608602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608577 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc088ed2-9869-4f39-aee7-abfab6bded71" containerName="payload-processing" Sep 09 17:49:26.608602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608586 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aa76aa3-9f57-4b09-80c2-23c3185327b8" containerName="maas-api" Sep 09 17:49:26.608602 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608592 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aa76aa3-9f57-4b09-80c2-23c3185327b8" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608609 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b00a756-6fc1-4043-bdd8-45599866e7a4" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608615 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b00a756-6fc1-4043-bdd8-45599866e7a4" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608624 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68411a2-7758-427b-973c-d0b04f3175b3" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608629 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68411a2-7758-427b-973c-d0b04f3175b3" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608638 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4403c382-221e-4a75-98fb-58c5d5e3089c" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608644 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4403c382-221e-4a75-98fb-58c5d5e3089c" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608734 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68411a2-7758-427b-973c-d0b04f3175b3" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608743 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aa76aa3-9f57-4b09-80c2-23c3185327b8" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608750 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f02d6c-644b-4910-a844-022f00f5e841" containerName="manager" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608757 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc088ed2-9869-4f39-aee7-abfab6bded71" containerName="payload-processing" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608764 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b00a756-6fc1-4043-bdd8-45599866e7a4" containerName="maas-api" Sep 09 17:49:26.608832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.608770 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4403c382-221e-4a75-98fb-58c5d5e3089c" containerName="maas-api" Sep 09 17:49:26.613959 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.612401 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:26.616783 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.616760 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9rfkr\"" Sep 09 17:49:26.623476 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.623224 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw"] Sep 09 17:49:26.738873 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.738847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxkh2\" (UniqueName: \"kubernetes.io/projected/ee199a24-7c2e-4e92-a928-2c102995e9db-kube-api-access-gxkh2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k6zxw\" (UID: \"ee199a24-7c2e-4e92-a928-2c102995e9db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:26.739038 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.738895 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee199a24-7c2e-4e92-a928-2c102995e9db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k6zxw\" (UID: \"ee199a24-7c2e-4e92-a928-2c102995e9db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:26.839893 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.839862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxkh2\" (UniqueName: \"kubernetes.io/projected/ee199a24-7c2e-4e92-a928-2c102995e9db-kube-api-access-gxkh2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k6zxw\" (UID: \"ee199a24-7c2e-4e92-a928-2c102995e9db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:26.840055 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.839907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee199a24-7c2e-4e92-a928-2c102995e9db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k6zxw\" (UID: \"ee199a24-7c2e-4e92-a928-2c102995e9db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:26.840213 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.840198 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee199a24-7c2e-4e92-a928-2c102995e9db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k6zxw\" (UID: \"ee199a24-7c2e-4e92-a928-2c102995e9db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:26.848238 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.848212 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxkh2\" (UniqueName: \"kubernetes.io/projected/ee199a24-7c2e-4e92-a928-2c102995e9db-kube-api-access-gxkh2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-k6zxw\" (UID: \"ee199a24-7c2e-4e92-a928-2c102995e9db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:26.928343 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:26.928284 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:27.207159 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:27.207135 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw"] Sep 09 17:49:27.210148 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:49:27.210118 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee199a24_7c2e_4e92_a928_2c102995e9db.slice/crio-3ac5ee100becdcaaff588c395ec0eb4d6597f14eec52399636a7142e60cb88e1 WatchSource:0}: Error finding container 3ac5ee100becdcaaff588c395ec0eb4d6597f14eec52399636a7142e60cb88e1: Status 404 returned error can't find the container with id 3ac5ee100becdcaaff588c395ec0eb4d6597f14eec52399636a7142e60cb88e1 Sep 09 17:49:27.434633 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:27.434593 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" event={"ID":"ee199a24-7c2e-4e92-a928-2c102995e9db","Type":"ContainerStarted","Data":"405f86df0aa587b8935d3dcef999e871b24dbbbce9869d37d79d64d2bbdba9ce"} Sep 09 17:49:27.434820 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:27.434760 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:27.434820 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:27.434787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" event={"ID":"ee199a24-7c2e-4e92-a928-2c102995e9db","Type":"ContainerStarted","Data":"3ac5ee100becdcaaff588c395ec0eb4d6597f14eec52399636a7142e60cb88e1"} Sep 09 17:49:27.453679 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:27.453622 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" podStartSLOduration=1.453606508 podStartE2EDuration="1.453606508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:49:27.452283017 +0000 UTC m=+1961.800644492" watchObservedRunningTime="2026-09-09 17:49:27.453606508 +0000 UTC m=+1961.801967980" Sep 09 17:49:38.443353 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:38.443324 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-k6zxw" Sep 09 17:49:49.087186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:49.087152 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:49:57.090405 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:49:57.090371 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:50:02.924096 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:50:02.924058 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:50:23.590217 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:50:23.590179 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:50:48.580494 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:50:48.580459 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:50:57.781281 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:50:57.781243 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:51:05.489799 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:05.489763 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:51:11.079158 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.079078 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:51:11.543671 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.543631 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf"] Sep 09 17:51:11.547698 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.547681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.550301 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.550277 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 17:51:11.550426 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.550344 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bxbtq\"" Sep 09 17:51:11.551460 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.551440 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 17:51:11.551571 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.551559 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 17:51:11.551632 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.551619 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 17:51:11.561569 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.561542 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf"] Sep 09 17:51:11.563468 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.563445 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k"] Sep 09 17:51:11.566995 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.566976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.578194 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.578168 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k"] Sep 09 17:51:11.623996 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.623961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b05431d-98c4-48b6-8e23-34c94a584d56-openshift-service-ca\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.624199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.624006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthcz\" (UniqueName: \"kubernetes.io/projected/89773af0-57cd-4c8f-9d6a-ee1125c17442-kube-api-access-zthcz\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.624199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.624077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-metrics-certs\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.624199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.624104 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-maas-api-tls\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.624199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.624167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfl6\" (UniqueName: \"kubernetes.io/projected/6b05431d-98c4-48b6-8e23-34c94a584d56-kube-api-access-5mfl6\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.624199 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.624198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b05431d-98c4-48b6-8e23-34c94a584d56-metrics-certs\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.624436 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.624235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b05431d-98c4-48b6-8e23-34c94a584d56-maas-api-tls\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.624436 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.624320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89773af0-57cd-4c8f-9d6a-ee1125c17442-openshift-service-ca\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.725306 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.725270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfl6\" (UniqueName: \"kubernetes.io/projected/6b05431d-98c4-48b6-8e23-34c94a584d56-kube-api-access-5mfl6\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.725306 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.725314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b05431d-98c4-48b6-8e23-34c94a584d56-metrics-certs\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.725559 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.725426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b05431d-98c4-48b6-8e23-34c94a584d56-maas-api-tls\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.725559 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.725465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89773af0-57cd-4c8f-9d6a-ee1125c17442-openshift-service-ca\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.725559 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.725512 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b05431d-98c4-48b6-8e23-34c94a584d56-openshift-service-ca\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.725559 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.725531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthcz\" (UniqueName: \"kubernetes.io/projected/89773af0-57cd-4c8f-9d6a-ee1125c17442-kube-api-access-zthcz\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.725798 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.725599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-metrics-certs\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.725798 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.725631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-maas-api-tls\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.726335 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.726311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89773af0-57cd-4c8f-9d6a-ee1125c17442-openshift-service-ca\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.726442 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.726311 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b05431d-98c4-48b6-8e23-34c94a584d56-openshift-service-ca\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.728265 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.728246 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b05431d-98c4-48b6-8e23-34c94a584d56-maas-api-tls\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.728356 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.728247 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-metrics-certs\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.728458 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.728436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-maas-api-tls\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.728520 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.728436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b05431d-98c4-48b6-8e23-34c94a584d56-metrics-certs\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.741123 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.741095 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfl6\" (UniqueName: \"kubernetes.io/projected/6b05431d-98c4-48b6-8e23-34c94a584d56-kube-api-access-5mfl6\") pod \"maas-api-7cfd4ff64b-kd79k\" (UID: \"6b05431d-98c4-48b6-8e23-34c94a584d56\") " pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:11.742366 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.742337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthcz\" (UniqueName: \"kubernetes.io/projected/89773af0-57cd-4c8f-9d6a-ee1125c17442-kube-api-access-zthcz\") pod \"maas-api-8458467cb6-dm9zf\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.859933 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.859849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:11.877701 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:11.877676 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:12.131896 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:12.131822 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf"] Sep 09 17:51:12.135735 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:51:12.135696 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89773af0_57cd_4c8f_9d6a_ee1125c17442.slice/crio-d36fc217d09e93fe639056f914adb94610d163c05a8b78b4cfa0a32d03f1bf4d WatchSource:0}: Error finding container d36fc217d09e93fe639056f914adb94610d163c05a8b78b4cfa0a32d03f1bf4d: Status 404 returned error can't find the container with id d36fc217d09e93fe639056f914adb94610d163c05a8b78b4cfa0a32d03f1bf4d Sep 09 17:51:12.137091 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:12.137073 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 17:51:12.157746 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:12.157725 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k"] Sep 09 17:51:12.159237 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:51:12.159218 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b05431d_98c4_48b6_8e23_34c94a584d56.slice/crio-335979bd323887f93cb3cf52566e151fe23b1c1b2dab6d12667d39e3fa4e06b5 WatchSource:0}: Error finding container 335979bd323887f93cb3cf52566e151fe23b1c1b2dab6d12667d39e3fa4e06b5: Status 404 returned error can't find the container with id 335979bd323887f93cb3cf52566e151fe23b1c1b2dab6d12667d39e3fa4e06b5 Sep 09 17:51:13.024925 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:13.024878 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" event={"ID":"89773af0-57cd-4c8f-9d6a-ee1125c17442","Type":"ContainerStarted","Data":"99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4"} Sep 09 17:51:13.025126 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:13.024962 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" event={"ID":"89773af0-57cd-4c8f-9d6a-ee1125c17442","Type":"ContainerStarted","Data":"d36fc217d09e93fe639056f914adb94610d163c05a8b78b4cfa0a32d03f1bf4d"} Sep 09 17:51:13.025126 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:13.025007 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:13.026011 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:13.025983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" event={"ID":"6b05431d-98c4-48b6-8e23-34c94a584d56","Type":"ContainerStarted","Data":"335979bd323887f93cb3cf52566e151fe23b1c1b2dab6d12667d39e3fa4e06b5"} Sep 09 17:51:13.044276 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:13.044230 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" podStartSLOduration=1.556673665 podStartE2EDuration="2.044215732s" totalImagesPullingTime="487.542067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:51:13.041532816 +0000 UTC m=+2067.389894314" watchObservedRunningTime="2026-09-09 17:51:13.044215732 +0000 UTC m=+2067.392577204" Sep 09 17:51:14.033759 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:14.033699 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" event={"ID":"6b05431d-98c4-48b6-8e23-34c94a584d56","Type":"ContainerStarted","Data":"1a30ead3e92b470d1c917fa06f740d75fdde053167ebce8f92f28c9872cbea1b"} Sep 09 17:51:14.054241 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:14.054189 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" podStartSLOduration=1.394939327 podStartE2EDuration="3.054172931s" totalImagesPullingTime="1.659233604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 17:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:51:14.052500449 +0000 UTC m=+2068.400861920" watchObservedRunningTime="2026-09-09 17:51:14.054172931 +0000 UTC m=+2068.402534403" Sep 09 17:51:15.039980 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:15.039942 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:19.039445 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:19.039414 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:19.087223 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:19.087188 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:51:21.050952 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.050923 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7cfd4ff64b-kd79k" Sep 09 17:51:21.096357 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.096315 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf"] Sep 09 17:51:21.096632 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.096559 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" podUID="89773af0-57cd-4c8f-9d6a-ee1125c17442" containerName="maas-api" containerID="cri-o://99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4" gracePeriod=30 Sep 09 17:51:21.343169 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.343143 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:21.420287 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.420254 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-metrics-certs\") pod \"89773af0-57cd-4c8f-9d6a-ee1125c17442\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " Sep 09 17:51:21.420483 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.420317 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89773af0-57cd-4c8f-9d6a-ee1125c17442-openshift-service-ca\") pod \"89773af0-57cd-4c8f-9d6a-ee1125c17442\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " Sep 09 17:51:21.420483 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.420346 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zthcz\" (UniqueName: \"kubernetes.io/projected/89773af0-57cd-4c8f-9d6a-ee1125c17442-kube-api-access-zthcz\") pod \"89773af0-57cd-4c8f-9d6a-ee1125c17442\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " Sep 09 17:51:21.420483 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.420379 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-maas-api-tls\") pod \"89773af0-57cd-4c8f-9d6a-ee1125c17442\" (UID: \"89773af0-57cd-4c8f-9d6a-ee1125c17442\") " Sep 09 17:51:21.420758 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.420732 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89773af0-57cd-4c8f-9d6a-ee1125c17442-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89773af0-57cd-4c8f-9d6a-ee1125c17442" (UID: "89773af0-57cd-4c8f-9d6a-ee1125c17442"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 17:51:21.422517 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.422482 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89773af0-57cd-4c8f-9d6a-ee1125c17442" (UID: "89773af0-57cd-4c8f-9d6a-ee1125c17442"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:51:21.422614 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.422536 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89773af0-57cd-4c8f-9d6a-ee1125c17442" (UID: "89773af0-57cd-4c8f-9d6a-ee1125c17442"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 17:51:21.422614 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.422606 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89773af0-57cd-4c8f-9d6a-ee1125c17442-kube-api-access-zthcz" (OuterVolumeSpecName: "kube-api-access-zthcz") pod "89773af0-57cd-4c8f-9d6a-ee1125c17442" (UID: "89773af0-57cd-4c8f-9d6a-ee1125c17442"). InnerVolumeSpecName "kube-api-access-zthcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 17:51:21.521968 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.521931 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-metrics-certs\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:51:21.521968 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.521961 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89773af0-57cd-4c8f-9d6a-ee1125c17442-openshift-service-ca\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:51:21.521968 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.521970 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zthcz\" (UniqueName: \"kubernetes.io/projected/89773af0-57cd-4c8f-9d6a-ee1125c17442-kube-api-access-zthcz\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:51:21.522206 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:21.521980 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89773af0-57cd-4c8f-9d6a-ee1125c17442-maas-api-tls\") on node \"ip-10-0-128-235.ec2.internal\" DevicePath \"\"" Sep 09 17:51:22.080143 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.080108 2580 generic.go:356] "Generic (PLEG): container finished" podID="89773af0-57cd-4c8f-9d6a-ee1125c17442" containerID="99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4" exitCode=0 Sep 09 17:51:22.080594 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.080156 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" Sep 09 17:51:22.080594 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.080174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" event={"ID":"89773af0-57cd-4c8f-9d6a-ee1125c17442","Type":"ContainerDied","Data":"99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4"} Sep 09 17:51:22.080594 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.080217 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf" event={"ID":"89773af0-57cd-4c8f-9d6a-ee1125c17442","Type":"ContainerDied","Data":"d36fc217d09e93fe639056f914adb94610d163c05a8b78b4cfa0a32d03f1bf4d"} Sep 09 17:51:22.080594 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.080237 2580 scope.go:117] "RemoveContainer" containerID="99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4" Sep 09 17:51:22.089612 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.089592 2580 scope.go:117] "RemoveContainer" containerID="99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4" Sep 09 17:51:22.089904 ip-10-0-128-235 kubenswrapper[2580]: E0909 17:51:22.089885 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4\": container with ID starting with 99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4 not found: ID does not exist" containerID="99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4" Sep 09 17:51:22.089987 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.089909 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4"} err="failed to get container status \"99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4\": rpc error: code = NotFound desc = could not find container \"99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4\": container with ID starting with 99f9f500af4e7efb86f3017e7f67217a33b37b25c830dd30afafbb9fbffed3d4 not found: ID does not exist" Sep 09 17:51:22.100821 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.100798 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf"] Sep 09 17:51:22.104502 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.104478 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8458467cb6-dm9zf"] Sep 09 17:51:22.139269 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:22.139232 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89773af0-57cd-4c8f-9d6a-ee1125c17442" path="/var/lib/kubelet/pods/89773af0-57cd-4c8f-9d6a-ee1125c17442/volumes" Sep 09 17:51:26.084235 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:26.084196 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:51:31.982150 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:31.982114 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:51:38.581566 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:38.581525 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tzkqv"] Sep 09 17:51:46.373685 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:46.373656 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:51:46.377131 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:46.377106 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:51:46.381642 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:46.381623 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:51:46.384812 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:46.384794 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:51:47.270239 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:47.270203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n5slf_529ad62f-ef17-4186-8b23-45a601d9bfef/manager/0.log" Sep 09 17:51:47.397200 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:47.397160 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-szp6l_19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b/manager/0.log" Sep 09 17:51:48.163821 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:48.163779 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cdfc9465f-zk66n_0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c/copy-manifests/0.log" Sep 09 17:51:48.177779 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:48.177739 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cdfc9465f-zk66n_0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c/manager/0.log" Sep 09 17:51:48.533649 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:48.533605 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7cfd4ff64b-kd79k_6b05431d-98c4-48b6-8e23-34c94a584d56/maas-api/0.log" Sep 09 17:51:49.804031 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:49.804002 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-qcjwm_49626bf1-e673-452a-8f0f-5f84adf7a9d3/manager/0.log" Sep 09 17:51:49.929502 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:49.929470 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vpb4q_db0b660e-bbbf-44cb-9ddc-aa196c8ceddd/manager/0.log" Sep 09 17:51:50.063501 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:50.063420 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-m5rs2_6c4eee49-a401-44f5-9d03-9a87269ba67c/kuadrant-console-plugin/0.log" Sep 09 17:51:50.193810 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:50.193781 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-k6zxw_ee199a24-7c2e-4e92-a928-2c102995e9db/manager/0.log" Sep 09 17:51:50.333093 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:50.333016 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tzkqv_14ad9c8c-3e0e-4c73-8462-8919f884220f/limitador/0.log" Sep 09 17:51:50.451497 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:50.451470 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-swb96_9b29873e-daf4-4b8c-93a6-77a0b0464522/manager/0.log" Sep 09 17:51:51.050122 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:51.050093 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d7468889-l5s9n_b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a/kube-auth-proxy/0.log" Sep 09 17:51:51.306459 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:51.306375 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bwbv4_e078a357-0326-49bd-901c-31cbcaad0861/istio-proxy/1.log" Sep 09 17:51:52.028614 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:52.028585 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-fwzsb_3edaa1de-d71a-42e4-a3ec-cd79b51a5e97/main/0.log" Sep 09 17:51:52.370887 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:51:52.370806 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-vhptk_8dc5fca2-0b1d-412b-bb3a-e7f54e825956/main/0.log" Sep 09 17:52:02.528539 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:02.528504 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mt749_fb607612-0bb9-4046-9e6f-c499d4eb1564/global-pull-secret-syncer/0.log" Sep 09 17:52:02.674076 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:02.674041 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vzr46_e40de956-6de3-45a1-9117-bd61ee8a6bc1/konnectivity-agent/0.log" Sep 09 17:52:02.695665 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:02.695627 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-235.ec2.internal_f4f8f7c2419a6847f0c2e2dd84ea218d/haproxy/0.log" Sep 09 17:52:07.854629 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:07.854598 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-qcjwm_49626bf1-e673-452a-8f0f-5f84adf7a9d3/manager/0.log" Sep 09 17:52:07.882141 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:07.882104 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vpb4q_db0b660e-bbbf-44cb-9ddc-aa196c8ceddd/manager/0.log" Sep 09 17:52:07.918817 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:07.918784 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-m5rs2_6c4eee49-a401-44f5-9d03-9a87269ba67c/kuadrant-console-plugin/0.log" Sep 09 17:52:07.957118 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:07.957091 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-k6zxw_ee199a24-7c2e-4e92-a928-2c102995e9db/manager/0.log" Sep 09 17:52:07.984885 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:07.984861 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tzkqv_14ad9c8c-3e0e-4c73-8462-8919f884220f/limitador/0.log" Sep 09 17:52:08.033059 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:08.033029 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-swb96_9b29873e-daf4-4b8c-93a6-77a0b0464522/manager/0.log" Sep 09 17:52:09.390819 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.390794 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecdd432-bb98-41f5-b6f8-e603f3eb601a/alertmanager/0.log" Sep 09 17:52:09.412290 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.412263 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecdd432-bb98-41f5-b6f8-e603f3eb601a/config-reloader/0.log" Sep 09 17:52:09.436029 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.436010 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecdd432-bb98-41f5-b6f8-e603f3eb601a/kube-rbac-proxy-web/0.log" Sep 09 17:52:09.455407 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.455392 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecdd432-bb98-41f5-b6f8-e603f3eb601a/kube-rbac-proxy/0.log" Sep 09 17:52:09.477701 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.477683 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecdd432-bb98-41f5-b6f8-e603f3eb601a/kube-rbac-proxy-metric/0.log" Sep 09 17:52:09.497036 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.497019 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecdd432-bb98-41f5-b6f8-e603f3eb601a/prom-label-proxy/0.log" Sep 09 17:52:09.517098 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.517081 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecdd432-bb98-41f5-b6f8-e603f3eb601a/init-config-reloader/0.log" Sep 09 17:52:09.557816 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.557797 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-nh2hz_930aa645-7318-4c90-850d-560c1430ab0f/cluster-monitoring-operator/0.log" Sep 09 17:52:09.581346 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.581318 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-h5zqn_96f1a4da-722a-4a7f-913d-e03bfd93f51a/kube-state-metrics/0.log" Sep 09 17:52:09.600815 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.600784 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-h5zqn_96f1a4da-722a-4a7f-913d-e03bfd93f51a/kube-rbac-proxy-main/0.log" Sep 09 17:52:09.620172 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.620148 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-h5zqn_96f1a4da-722a-4a7f-913d-e03bfd93f51a/kube-rbac-proxy-self/0.log" Sep 09 17:52:09.702040 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.701957 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4dqph_de31f071-6ade-4899-a46a-7ed675ee649b/node-exporter/0.log" Sep 09 17:52:09.737086 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.737059 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4dqph_de31f071-6ade-4899-a46a-7ed675ee649b/kube-rbac-proxy/0.log" Sep 09 17:52:09.767439 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.767414 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4dqph_de31f071-6ade-4899-a46a-7ed675ee649b/init-textfile/0.log" Sep 09 17:52:09.974116 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:09.974042 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2jrfd_bebc1406-5ed1-4286-9f7f-3cb9158e09b7/kube-rbac-proxy-main/0.log" Sep 09 17:52:10.004311 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.004288 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2jrfd_bebc1406-5ed1-4286-9f7f-3cb9158e09b7/kube-rbac-proxy-self/0.log" Sep 09 17:52:10.024897 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.024873 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-2jrfd_bebc1406-5ed1-4286-9f7f-3cb9158e09b7/openshift-state-metrics/0.log" Sep 09 17:52:10.276765 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.276734 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-zrqk6_45c9e5f4-26c4-40f4-8bae-e52c0271909a/prometheus-operator-admission-webhook/0.log" Sep 09 17:52:10.308020 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.308001 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d47778895-8gfx6_0badb01a-0d3b-4ecc-95f6-2f81e94219f6/telemeter-client/0.log" Sep 09 17:52:10.327768 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.327706 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d47778895-8gfx6_0badb01a-0d3b-4ecc-95f6-2f81e94219f6/reload/0.log" Sep 09 17:52:10.350641 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.350622 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d47778895-8gfx6_0badb01a-0d3b-4ecc-95f6-2f81e94219f6/kube-rbac-proxy/0.log" Sep 09 17:52:10.379952 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.379933 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c85b85d77-lpq9q_8b58e555-f3b9-42cc-9e26-e822205efaf0/thanos-query/0.log" Sep 09 17:52:10.401159 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.401133 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c85b85d77-lpq9q_8b58e555-f3b9-42cc-9e26-e822205efaf0/kube-rbac-proxy-web/0.log" Sep 09 17:52:10.421659 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.421642 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c85b85d77-lpq9q_8b58e555-f3b9-42cc-9e26-e822205efaf0/kube-rbac-proxy/0.log" Sep 09 17:52:10.441508 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.441490 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c85b85d77-lpq9q_8b58e555-f3b9-42cc-9e26-e822205efaf0/prom-label-proxy/0.log" Sep 09 17:52:10.461949 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.461926 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c85b85d77-lpq9q_8b58e555-f3b9-42cc-9e26-e822205efaf0/kube-rbac-proxy-rules/0.log" Sep 09 17:52:10.483487 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:10.483465 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c85b85d77-lpq9q_8b58e555-f3b9-42cc-9e26-e822205efaf0/kube-rbac-proxy-metrics/0.log" Sep 09 17:52:12.488655 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:12.488627 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/1.log" Sep 09 17:52:12.493987 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:12.493950 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-z9596_9b4e0a01-e009-4a59-9826-b82707527015/console-operator/2.log" Sep 09 17:52:13.058669 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.058640 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-2nnll_0f4ede09-9816-432e-9f1f-652dd1ab3a5f/download-server/0.log" Sep 09 17:52:13.230626 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.230589 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr"] Sep 09 17:52:13.231235 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.231213 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89773af0-57cd-4c8f-9d6a-ee1125c17442" containerName="maas-api" Sep 09 17:52:13.231235 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.231236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="89773af0-57cd-4c8f-9d6a-ee1125c17442" containerName="maas-api" Sep 09 17:52:13.231411 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.231319 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="89773af0-57cd-4c8f-9d6a-ee1125c17442" containerName="maas-api" Sep 09 17:52:13.234657 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.234640 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.237170 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.237147 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k64x\"/\"kube-root-ca.crt\"" Sep 09 17:52:13.237284 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.237150 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8k64x\"/\"default-dockercfg-f5w8x\"" Sep 09 17:52:13.238265 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.238245 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k64x\"/\"openshift-service-ca.crt\"" Sep 09 17:52:13.243458 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.243436 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr"] Sep 09 17:52:13.327098 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.327006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-proc\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.327098 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.327048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-podres\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.327098 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.327077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lvzd\" (UniqueName: \"kubernetes.io/projected/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-kube-api-access-6lvzd\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.327342 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.327128 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-lib-modules\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.327342 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.327175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-sys\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428031 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.427999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-lib-modules\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.428072 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-sys\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.428112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-proc\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.428134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-podres\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.428160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lvzd\" (UniqueName: \"kubernetes.io/projected/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-kube-api-access-6lvzd\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.428175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-lib-modules\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.428205 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-sys\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428232 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.428214 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-proc\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.428480 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.428263 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-podres\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.436127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.436105 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lvzd\" (UniqueName: \"kubernetes.io/projected/2e5f07dc-8bd2-4c8f-af2f-f5849add6b64-kube-api-access-6lvzd\") pod \"perf-node-gather-daemonset-wrgpr\" (UID: \"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64\") " pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.545073 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.545041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:13.814937 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:13.814912 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr"] Sep 09 17:52:13.816972 ip-10-0-128-235 kubenswrapper[2580]: W0909 17:52:13.816945 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2e5f07dc_8bd2_4c8f_af2f_f5849add6b64.slice/crio-c136b3eab4fa6997a730eff42d5c90f42b094c7819258ffcb74e7e53f9823d55 WatchSource:0}: Error finding container c136b3eab4fa6997a730eff42d5c90f42b094c7819258ffcb74e7e53f9823d55: Status 404 returned error can't find the container with id c136b3eab4fa6997a730eff42d5c90f42b094c7819258ffcb74e7e53f9823d55 Sep 09 17:52:14.370000 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:14.369959 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" event={"ID":"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64","Type":"ContainerStarted","Data":"02fefc466f99d4e36f392fc82c80481fad61f451321046c08284f60ea7893de7"} Sep 09 17:52:14.370186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:14.370126 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:14.370186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:14.370163 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" event={"ID":"2e5f07dc-8bd2-4c8f-af2f-f5849add6b64","Type":"ContainerStarted","Data":"c136b3eab4fa6997a730eff42d5c90f42b094c7819258ffcb74e7e53f9823d55"} Sep 09 17:52:14.387513 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:14.387456 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" podStartSLOduration=1.38743956 podStartE2EDuration="1.38743956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 17:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 17:52:14.385204385 +0000 UTC m=+2128.733565856" watchObservedRunningTime="2026-09-09 17:52:14.38743956 +0000 UTC m=+2128.735801033" Sep 09 17:52:14.920959 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:14.920930 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8j8h9_b79ceb92-b128-4b3b-9fa5-64ae8bbb384f/dns/0.log" Sep 09 17:52:14.940516 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:14.940493 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8j8h9_b79ceb92-b128-4b3b-9fa5-64ae8bbb384f/kube-rbac-proxy/0.log" Sep 09 17:52:15.048772 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:15.048740 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bkxw4_07eab57a-6bbe-4919-8121-a5fb27de7f48/dns-node-resolver/0.log" Sep 09 17:52:15.651145 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:15.651117 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vcxxj_44fac9f1-d2e6-4397-bb60-47cf60343669/node-ca/0.log" Sep 09 17:52:16.905186 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:16.905152 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d7468889-l5s9n_b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a/kube-auth-proxy/0.log" Sep 09 17:52:16.962308 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:16.962270 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bwbv4_e078a357-0326-49bd-901c-31cbcaad0861/istio-proxy/0.log" Sep 09 17:52:16.970898 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:16.970876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bwbv4_e078a357-0326-49bd-901c-31cbcaad0861/istio-proxy/1.log" Sep 09 17:52:17.857785 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:17.857758 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2nh4_0376df71-4ec7-4127-94d1-42318c23a135/serve-healthcheck-canary/0.log" Sep 09 17:52:18.511154 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:18.511126 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wjqgn_bcd3c8d8-2c37-43ec-8cab-ed05b059bb75/kube-rbac-proxy/0.log" Sep 09 17:52:18.533094 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:18.533070 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wjqgn_bcd3c8d8-2c37-43ec-8cab-ed05b059bb75/exporter/0.log" Sep 09 17:52:18.552810 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:18.552787 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wjqgn_bcd3c8d8-2c37-43ec-8cab-ed05b059bb75/extractor/0.log" Sep 09 17:52:20.384539 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:20.384511 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8k64x/perf-node-gather-daemonset-wrgpr" Sep 09 17:52:20.636076 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:20.635988 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n5slf_529ad62f-ef17-4186-8b23-45a601d9bfef/manager/0.log" Sep 09 17:52:20.760147 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:20.760114 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-szp6l_19cebdd2-d398-40ba-bc4d-ab2f7a98ce5b/manager/0.log" Sep 09 17:52:21.103155 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:21.103125 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cdfc9465f-zk66n_0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c/manager/0.log" Sep 09 17:52:21.119878 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:21.119854 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cdfc9465f-zk66n_0a3a6fab-25ff-4c1f-ac31-0bc30fe1473c/copy-manifests/0.log" Sep 09 17:52:23.492059 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:23.492022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d7468889-l5s9n_b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a/kube-auth-proxy/0.log" Sep 09 17:52:23.533244 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:23.533215 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bwbv4_e078a357-0326-49bd-901c-31cbcaad0861/istio-proxy/0.log" Sep 09 17:52:23.542660 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:23.542625 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bwbv4_e078a357-0326-49bd-901c-31cbcaad0861/istio-proxy/1.log" Sep 09 17:52:25.500425 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:25.500394 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d7468889-l5s9n_b1c9b8dc-c002-4aeb-9c06-7f7ae199df7a/kube-auth-proxy/0.log" Sep 09 17:52:25.540447 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:25.540411 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bwbv4_e078a357-0326-49bd-901c-31cbcaad0861/istio-proxy/0.log" Sep 09 17:52:25.551257 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:25.551183 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-bwbv4_e078a357-0326-49bd-901c-31cbcaad0861/istio-proxy/1.log" Sep 09 17:52:27.514402 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:27.514367 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5647g_2d3aafe0-62ec-46e3-8d0b-aeb98d80060c/migrator/0.log" Sep 09 17:52:27.536930 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:27.536904 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-5647g_2d3aafe0-62ec-46e3-8d0b-aeb98d80060c/graceful-termination/0.log" Sep 09 17:52:27.875666 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:27.875589 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-m8sl5_d11da5db-6654-463b-8880-b7aade5a657f/kube-storage-version-migrator-operator/1.log" Sep 09 17:52:27.876565 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:27.876545 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-m8sl5_d11da5db-6654-463b-8880-b7aade5a657f/kube-storage-version-migrator-operator/0.log" Sep 09 17:52:29.000532 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.000492 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nr8h7_9bf0bee3-0ee6-4c63-b165-87ddd060ec71/kube-multus-additional-cni-plugins/0.log" Sep 09 17:52:29.019519 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.019493 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nr8h7_9bf0bee3-0ee6-4c63-b165-87ddd060ec71/egress-router-binary-copy/0.log" Sep 09 17:52:29.037947 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.037899 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nr8h7_9bf0bee3-0ee6-4c63-b165-87ddd060ec71/cni-plugins/0.log" Sep 09 17:52:29.056411 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.056390 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nr8h7_9bf0bee3-0ee6-4c63-b165-87ddd060ec71/bond-cni-plugin/0.log" Sep 09 17:52:29.077560 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.077532 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nr8h7_9bf0bee3-0ee6-4c63-b165-87ddd060ec71/routeoverride-cni/0.log" Sep 09 17:52:29.097127 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.097106 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nr8h7_9bf0bee3-0ee6-4c63-b165-87ddd060ec71/whereabouts-cni-bincopy/0.log" Sep 09 17:52:29.116832 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.116809 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nr8h7_9bf0bee3-0ee6-4c63-b165-87ddd060ec71/whereabouts-cni/0.log" Sep 09 17:52:29.346042 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.346017 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vrk7q_22eae431-e183-4dfb-a650-967703b13aea/kube-multus/0.log" Sep 09 17:52:29.408449 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.408422 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kkvpl_f9649a0b-cc7d-4eb0-9952-865e10002bd7/network-metrics-daemon/0.log" Sep 09 17:52:29.423205 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:29.423187 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kkvpl_f9649a0b-cc7d-4eb0-9952-865e10002bd7/kube-rbac-proxy/0.log" Sep 09 17:52:30.685407 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.685384 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-controller/0.log" Sep 09 17:52:30.700248 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.700225 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/1.log" Sep 09 17:52:30.700387 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.700257 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovn-acl-logging/0.log" Sep 09 17:52:30.719965 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.719946 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/kube-rbac-proxy-node/0.log" Sep 09 17:52:30.741150 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.741125 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 17:52:30.756540 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.756520 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/northd/0.log" Sep 09 17:52:30.776015 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.775991 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/nbdb/0.log" Sep 09 17:52:30.795416 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.795399 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/sbdb/0.log" Sep 09 17:52:30.846750 ip-10-0-128-235 kubenswrapper[2580]: I0909 17:52:30.846701 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-z4nmc_46c0b0f4-1521-4894-a946-d3c93fb83c39/ovnkube-controller/0.log"