Jul 22 12:38:46.718401 ip-10-0-137-88 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 22 12:38:46.718414 ip-10-0-137-88 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 22 12:38:46.718423 ip-10-0-137-88 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 22 12:38:46.718718 ip-10-0-137-88 systemd[1]: Failed to start Kubernetes Kubelet. Jul 22 12:38:56.316396 ip-10-0-137-88 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 22 12:38:56.316410 ip-10-0-137-88 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 18aa8ae05a044b03bfd3ccd809acb736 -- Jul 22 12:40:44.978647 ip-10-0-137-88 systemd[1]: Starting Kubernetes Kubelet... Jul 22 12:40:45.376625 ip-10-0-137-88 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:45.376625 ip-10-0-137-88 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 12:40:45.376625 ip-10-0-137-88 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:45.376625 ip-10-0-137-88 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 12:40:45.376625 ip-10-0-137-88 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:45.377237 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.376679 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 12:40:45.381206 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381190 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:45.381206 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381206 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381210 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381214 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381216 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381220 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381222 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381225 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381227 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381230 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381233 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381235 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381238 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381240 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381242 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381256 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381259 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381262 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381264 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381267 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381269 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:45.381272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381272 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381274 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381279 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381283 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381286 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381301 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381306 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381310 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381313 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381317 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381321 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381324 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381326 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381328 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381331 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381333 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381335 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381337 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381340 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:45.381759 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381344 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381347 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381350 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381353 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381355 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381359 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381364 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381366 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381369 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381372 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381374 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381376 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381379 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381381 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381384 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381386 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381389 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381392 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381394 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381397 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381399 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:45.382200 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381401 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381404 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381407 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381409 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381411 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381414 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381416 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381418 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381421 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381423 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381426 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381428 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381430 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381433 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381435 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381437 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381439 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381442 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381445 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:45.382707 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381447 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381450 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381452 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381454 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381457 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381459 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381462 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381465 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381468 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381470 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381472 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381475 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381477 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381479 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381483 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381485 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381489 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381492 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.381495 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382161 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:45.383186 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382167 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382170 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382173 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382175 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382178 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382180 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382183 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382185 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382187 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382190 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382192 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382195 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382198 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382200 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382205 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382208 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382211 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382214 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382218 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:45.383670 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382221 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382223 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382226 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382228 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382231 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382234 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382236 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382239 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382241 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382245 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382248 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382250 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382252 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382255 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382257 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382261 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382265 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382268 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382270 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:45.384126 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382273 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382276 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382279 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382282 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382284 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382302 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382305 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382309 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382312 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382314 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382317 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382319 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382322 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382324 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382327 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382329 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382332 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382334 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382336 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382339 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:45.384590 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382341 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382343 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382346 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382348 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382351 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382353 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382356 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382359 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382362 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382366 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382369 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382371 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382374 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382376 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382379 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382381 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382384 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382386 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382388 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:45.385068 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382391 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382394 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382396 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382398 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382401 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382404 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382406 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382408 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382411 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382413 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382416 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382418 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382420 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382423 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382425 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382427 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382430 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382432 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382434 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382437 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:45.385537 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382439 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382785 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382790 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382793 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382795 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382798 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382800 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382803 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382806 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382808 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382811 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382813 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382816 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382819 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382821 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382824 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382827 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382829 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382832 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382834 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:45.385994 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382837 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382839 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382841 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382844 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382846 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382848 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382851 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382853 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382856 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382858 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382872 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382874 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382877 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382879 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382882 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382885 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382887 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382890 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:45.386502 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382892 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382896 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382900 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382904 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382907 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382910 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382912 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382915 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382919 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382921 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382924 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382928 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382931 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382933 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382936 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382938 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382940 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382943 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382945 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:45.386921 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382947 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382950 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382952 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382954 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382956 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382959 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382961 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382964 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382967 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382969 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382972 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382974 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382976 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382978 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382981 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382983 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382986 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382988 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382991 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382993 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:45.387402 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382995 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.382998 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383001 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383003 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383006 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383010 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383012 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383015 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383017 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383019 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383022 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383024 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383027 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383029 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383032 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383034 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383036 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383039 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383041 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383044 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:45.387900 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383046 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383049 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383051 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383122 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383128 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383134 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383139 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383142 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383145 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383149 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383153 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383155 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383158 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383161 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383164 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383167 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383171 2576 flags.go:64] FLAG: --cgroup-root="" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383174 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383177 2576 flags.go:64] FLAG: --client-ca-file="" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383180 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383182 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383187 2576 flags.go:64] FLAG: --cluster-domain="" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383190 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 12:40:45.388373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383192 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383195 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383198 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383202 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383204 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383207 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383210 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383213 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383216 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383219 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383222 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383225 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383229 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383232 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383234 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383237 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383240 2576 flags.go:64] FLAG: --enable-server="true" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383242 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383247 2576 flags.go:64] FLAG: --event-burst="100" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383250 2576 flags.go:64] FLAG: --event-qps="50" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383252 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383255 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383257 2576 flags.go:64] FLAG: --eviction-hard="" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383265 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 12:40:45.388997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383268 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383272 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383274 2576 flags.go:64] FLAG: --eviction-soft="" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383277 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383279 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383282 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383285 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383299 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383302 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383305 2576 flags.go:64] FLAG: --feature-gates="" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383308 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383311 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383314 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383317 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383319 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383322 2576 flags.go:64] FLAG: --help="false" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383326 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383329 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383332 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383335 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383338 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383341 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383343 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 12:40:45.389647 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383346 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383348 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383351 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383353 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383356 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383359 2576 flags.go:64] FLAG: --kube-reserved="" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383361 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383364 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383366 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383369 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383372 2576 flags.go:64] FLAG: --lock-file="" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383375 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383377 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383380 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383385 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383387 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383390 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383392 2576 flags.go:64] FLAG: --logging-format="text" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383395 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383398 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383401 2576 flags.go:64] FLAG: --manifest-url="" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383403 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383408 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383410 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383414 2576 flags.go:64] FLAG: --max-pods="110" Jul 22 12:40:45.390213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383418 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383421 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383423 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383426 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383429 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383432 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383435 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383441 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383444 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383447 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383450 2576 flags.go:64] FLAG: --pod-cidr="" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383453 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383457 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383460 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383463 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383465 2576 flags.go:64] FLAG: --port="10250" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383468 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383471 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02d61fa8f89a30b58" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383474 2576 flags.go:64] FLAG: --qos-reserved="" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383477 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383480 2576 flags.go:64] FLAG: --register-node="true" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383482 2576 flags.go:64] FLAG: --register-with-taints="" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383486 2576 flags.go:64] FLAG: --registry-burst="10" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383488 2576 flags.go:64] FLAG: --registry-qps="5" Jul 22 12:40:45.390819 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383491 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383493 2576 flags.go:64] FLAG: --reserved-memory="" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383496 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383499 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383502 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383504 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383506 2576 flags.go:64] FLAG: --runonce="false" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383509 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383513 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383516 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383518 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383521 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383523 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383526 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383529 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383531 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383534 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383537 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383539 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383542 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383545 2576 flags.go:64] FLAG: --system-cgroups="" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383547 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383552 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383554 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383557 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 12:40:45.391381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383561 2576 flags.go:64] FLAG: --tls-min-version="" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383563 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383566 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383569 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383571 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383574 2576 flags.go:64] FLAG: --v="2" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383577 2576 flags.go:64] FLAG: --version="false" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383581 2576 flags.go:64] FLAG: --vmodule="" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383584 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.383587 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383710 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383714 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383717 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383720 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383723 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383727 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383730 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383733 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383735 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383738 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383740 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383743 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:45.391958 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383746 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383748 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383751 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383753 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383756 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383758 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383761 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383764 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383766 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383768 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383773 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383776 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383778 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383780 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383783 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383786 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383788 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383790 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383793 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383795 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:45.392521 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383798 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383801 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383804 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383807 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383810 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383813 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383816 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383818 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383820 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383823 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383825 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383827 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383829 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383832 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383834 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383837 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383839 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383841 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383843 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:45.392984 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383846 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383848 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383850 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383855 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383858 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383860 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383862 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383865 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383867 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383870 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383872 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383875 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383877 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383880 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383882 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383884 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383887 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383889 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383892 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383895 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:45.393442 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383898 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383902 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383904 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383907 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383909 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383912 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383914 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383916 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383919 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383921 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383924 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383926 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383930 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383933 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383935 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383939 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383942 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383945 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383947 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:45.393925 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383950 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383952 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383961 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383964 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383967 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383969 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383971 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383974 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.383976 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.384607 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.391078 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.391091 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.391783 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.391790 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391859 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:45.394394 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391863 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391866 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391869 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391871 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391874 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391877 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391879 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391882 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391884 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391886 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391889 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391891 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391893 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391896 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391898 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391900 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391904 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391906 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391908 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391917 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:45.394805 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391919 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391922 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391924 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391928 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391932 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391935 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391937 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391939 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391943 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391946 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391948 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391951 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391953 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391956 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391958 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391961 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391964 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391967 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391969 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:45.395261 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391971 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391974 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391976 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391979 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391982 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391984 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391987 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391991 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391994 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391997 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.391999 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392002 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392004 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392012 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392014 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392017 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392019 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392021 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392024 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392026 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:45.395747 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392028 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392032 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392034 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392037 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392039 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392041 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392044 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392046 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392048 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392051 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392053 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392056 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392061 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392064 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392066 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392068 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392070 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392073 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392075 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392078 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:45.396230 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392080 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392082 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392084 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392087 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392089 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392091 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392099 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392102 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392104 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392107 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392110 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392113 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392116 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392119 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392121 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392124 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392127 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392129 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:45.396711 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392132 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.392137 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392277 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392283 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392286 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392309 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392313 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392316 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392318 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392321 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392323 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392326 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392328 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392331 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:45.397133 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392333 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392343 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392346 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392348 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392350 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392353 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392355 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392363 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392365 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392368 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392370 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392372 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392374 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392377 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392379 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392382 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392384 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392386 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392390 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:45.397480 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392393 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392396 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392398 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392401 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392404 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392406 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392409 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392411 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392413 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392415 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392418 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392420 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392422 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392424 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392427 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392430 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392432 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392434 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392436 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392439 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:45.397923 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392442 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392451 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392454 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392456 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392459 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392461 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392464 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392466 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392469 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392471 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392473 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392476 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392478 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392480 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392482 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392485 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392487 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392489 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392492 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392494 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:45.398391 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392496 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392499 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392501 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392504 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392506 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392508 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392511 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392513 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392515 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392518 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392520 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392522 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392525 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392527 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392535 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392538 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392540 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392543 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392545 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392547 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:45.398838 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392549 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392552 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392554 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392556 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392559 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392561 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392563 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:45.392566 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.392570 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.392661 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.394511 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.396072 2576 server.go:1019] "Starting client certificate rotation" Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.396190 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 12:40:45.399303 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.396227 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 12:40:45.400272 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.400260 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 12:40:45.401914 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.401901 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 12:40:45.410371 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.410350 2576 log.go:25] "Validated CRI v1 runtime API" Jul 22 12:40:45.412484 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.412467 2576 log.go:25] "Validated CRI v1 image API" Jul 22 12:40:45.413828 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.413814 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 12:40:45.416173 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.416144 2576 fs.go:135] Filesystem UUIDs: map[51a8300e-2802-4d9c-8111-f1c961aea52d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 83a4c7a2-295b-49a7-afc0-5d3d21fb178a:/dev/nvme0n1p4] Jul 22 12:40:45.416243 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.416172 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 12:40:45.421695 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.421584 2576 manager.go:216] Machine: {Timestamp:2026-07-22 12:40:45.420415112 +0000 UTC m=+0.328852569 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200696 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b6b07087531998cebe87679b933e2 SystemUUID:ec2b6b07-0875-3199-8ceb-e87679b933e2 BootID:18aa8ae0-5a04-4b03-bfd3-ccd809acb736 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:22:e2:f9:b4:d3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:22:e2:f9:b4:d3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:60:87:b4:b4:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 12:40:45.421695 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.421686 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 12:40:45.421847 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.421774 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 12:40:45.424646 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.424618 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 12:40:45.424773 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.424649 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 12:40:45.426067 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.426056 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 12:40:45.426106 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.426070 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 12:40:45.426106 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.426078 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 12:40:45.426556 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.426546 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 12:40:45.427611 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.427601 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 12:40:45.430099 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430088 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 22 12:40:45.430238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430228 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 12:40:45.430277 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430259 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 22 12:40:45.430325 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430280 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 12:40:45.430325 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430309 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 12:40:45.430325 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430317 2576 kubelet.go:398] "Adding apiserver pod source" Jul 22 12:40:45.430325 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430326 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 12:40:45.430663 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430651 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 12:40:45.430700 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.430672 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 12:40:45.431502 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.431477 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 12:40:45.433115 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.433102 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 12:40:45.435149 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.435137 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 12:40:45.435188 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.435166 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 12:40:45.436199 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436188 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 12:40:45.436230 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436205 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 12:40:45.436230 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436211 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 12:40:45.436230 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436216 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 12:40:45.436230 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436227 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 12:40:45.436356 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436237 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 12:40:45.436356 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436248 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 12:40:45.436356 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436253 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 12:40:45.436356 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436259 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 12:40:45.436356 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436277 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 12:40:45.436356 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436302 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 12:40:45.436356 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.436309 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 12:40:45.437041 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.437031 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 12:40:45.437072 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.437042 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 12:40:45.440169 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.440145 2576 server.go:1268] "Started kubelet" Jul 22 12:40:45.440270 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.440195 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 12:40:45.440352 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.440278 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 12:40:45.440419 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.440403 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 12:40:45.440559 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.440534 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 12:40:45.440605 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.440536 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 12:40:45.440680 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.440663 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 12:40:45.441654 ip-10-0-137-88 systemd[1]: Started Kubernetes Kubelet. Jul 22 12:40:45.441817 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.441788 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 12:40:45.447232 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.447211 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 22 12:40:45.447636 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.447617 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 12:40:45.450530 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.449475 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-88.ec2.internal.18c49d103e8e2fa1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-88.ec2.internal,UID:ip-10-0-137-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-88.ec2.internal,},FirstTimestamp:2026-07-22 12:40:45.440102305 +0000 UTC m=+0.348539762,LastTimestamp:2026-07-22 12:40:45.440102305 +0000 UTC m=+0.348539762,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-88.ec2.internal,}" Jul 22 12:40:45.451964 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.451951 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 12:40:45.452033 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.451965 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 12:40:45.452082 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.452047 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:45.452635 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.452485 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 12:40:45.452635 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.452502 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 12:40:45.452635 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.452594 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 12:40:45.452635 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.452603 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 22 12:40:45.452635 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.452614 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 22 12:40:45.453105 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.452954 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 12:40:45.453105 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.452967 2576 factory.go:55] Registering systemd factory Jul 22 12:40:45.453105 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.452975 2576 factory.go:223] Registration of the systemd container factory successfully Jul 22 12:40:45.453430 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.453411 2576 factory.go:153] Registering CRI-O factory Jul 22 12:40:45.453497 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.453435 2576 factory.go:223] Registration of the crio container factory successfully Jul 22 12:40:45.453497 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.453459 2576 factory.go:103] Registering Raw factory Jul 22 12:40:45.453497 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.453471 2576 manager.go:1196] Started watching for new ooms in manager Jul 22 12:40:45.453886 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.453855 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 12:40:45.453967 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.453894 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 12:40:45.454086 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.454073 2576 manager.go:349] Starting recovery of all containers Jul 22 12:40:45.465005 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.464990 2576 manager.go:354] Recovery completed Jul 22 12:40:45.468516 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.468504 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:45.470508 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.470408 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:45.470566 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.470528 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:45.470566 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.470538 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:45.471044 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.471029 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 12:40:45.471044 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.471043 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 12:40:45.471145 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.471049 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:45.471145 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.471078 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 22 12:40:45.471736 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.471721 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:45.471799 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.471745 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:45.471799 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.471757 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:45.472150 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.472139 2576 policy_none.go:49] "None policy: Start" Jul 22 12:40:45.472186 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.472157 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 12:40:45.472186 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.472169 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 12:40:45.473020 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.473010 2576 policy_none.go:47] "Start" Jul 22 12:40:45.481633 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.481573 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-88.ec2.internal.18c49d10405e1a81 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-88.ec2.internal,UID:ip-10-0-137-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-88.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-88.ec2.internal,},FirstTimestamp:2026-07-22 12:40:45.470505601 +0000 UTC m=+0.378943061,LastTimestamp:2026-07-22 12:40:45.470505601 +0000 UTC m=+0.378943061,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-88.ec2.internal,}" Jul 22 12:40:45.490179 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.490160 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nch6b" Jul 22 12:40:45.491848 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.491767 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-88.ec2.internal.18c49d10405e8752 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-88.ec2.internal,UID:ip-10-0-137-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-88.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-88.ec2.internal,},FirstTimestamp:2026-07-22 12:40:45.470533458 +0000 UTC m=+0.378970917,LastTimestamp:2026-07-22 12:40:45.470533458 +0000 UTC m=+0.378970917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-88.ec2.internal,}" Jul 22 12:40:45.500739 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.500714 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nch6b" Jul 22 12:40:45.501840 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.501780 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-88.ec2.internal.18c49d10405ea9ae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-88.ec2.internal,UID:ip-10-0-137-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-88.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-88.ec2.internal,},FirstTimestamp:2026-07-22 12:40:45.470542254 +0000 UTC m=+0.378979711,LastTimestamp:2026-07-22 12:40:45.470542254 +0000 UTC m=+0.378979711,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-88.ec2.internal,}" Jul 22 12:40:45.503824 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.503810 2576 manager.go:337] "Starting Device Plugin manager" Jul 22 12:40:45.503891 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.503837 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 12:40:45.503891 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.503847 2576 server.go:87] "Starting device plugin registration server" Jul 22 12:40:45.504081 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.504066 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 12:40:45.504181 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.504082 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 12:40:45.504240 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.504200 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 12:40:45.504311 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.504284 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 12:40:45.504358 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.504325 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 12:40:45.504780 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.504759 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 12:40:45.504840 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.504812 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:45.601429 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.601403 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 12:40:45.602693 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.602677 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 12:40:45.602745 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.602721 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 12:40:45.602745 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.602742 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 12:40:45.602830 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.602778 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 12:40:45.604167 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.604145 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:45.605042 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.605020 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:45.605112 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.605053 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:45.605112 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.605063 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:45.605112 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.605085 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.605429 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.605414 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:45.616395 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.616378 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.616455 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.616400 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-88.ec2.internal\": node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:45.639467 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.639425 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:45.703398 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.703378 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal"] Jul 22 12:40:45.703455 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.703428 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:45.704736 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.704721 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:45.704801 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.704752 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:45.704801 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.704762 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:45.705879 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.705868 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:45.705978 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.705964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.706030 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.705991 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:45.706666 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.706651 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:45.706745 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.706681 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:45.706745 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.706694 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:45.706745 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.706702 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:45.706745 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.706722 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:45.706745 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.706733 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:45.708328 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.708315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.708377 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.708337 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:45.709046 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.709026 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:45.709106 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.709055 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:45.709106 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.709069 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:45.736767 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.736750 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-88.ec2.internal\" not found" node="ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.739491 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.739472 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:45.741015 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.741001 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-88.ec2.internal\" not found" node="ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.754226 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.754200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c69c0cdb4de82ba0de3b9240fc59ceea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal\" (UID: \"c69c0cdb4de82ba0de3b9240fc59ceea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.754310 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.754229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c69c0cdb4de82ba0de3b9240fc59ceea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal\" (UID: \"c69c0cdb4de82ba0de3b9240fc59ceea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.754310 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.754250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349801908a8762aee344d69fa297f1e3-config\") pod \"kube-apiserver-proxy-ip-10-0-137-88.ec2.internal\" (UID: \"349801908a8762aee344d69fa297f1e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.839918 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.839899 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:45.855234 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.855212 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c69c0cdb4de82ba0de3b9240fc59ceea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal\" (UID: \"c69c0cdb4de82ba0de3b9240fc59ceea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.855333 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.855244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c69c0cdb4de82ba0de3b9240fc59ceea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal\" (UID: \"c69c0cdb4de82ba0de3b9240fc59ceea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.855333 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.855272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349801908a8762aee344d69fa297f1e3-config\") pod \"kube-apiserver-proxy-ip-10-0-137-88.ec2.internal\" (UID: \"349801908a8762aee344d69fa297f1e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.855427 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.855337 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/349801908a8762aee344d69fa297f1e3-config\") pod \"kube-apiserver-proxy-ip-10-0-137-88.ec2.internal\" (UID: \"349801908a8762aee344d69fa297f1e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.855427 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.855341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c69c0cdb4de82ba0de3b9240fc59ceea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal\" (UID: \"c69c0cdb4de82ba0de3b9240fc59ceea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.855427 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:45.855341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c69c0cdb4de82ba0de3b9240fc59ceea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal\" (UID: \"c69c0cdb4de82ba0de3b9240fc59ceea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:45.940605 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:45.940553 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:46.040073 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.040043 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:46.041114 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:46.041099 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:46.044505 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.043678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" Jul 22 12:40:46.142102 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:46.142083 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:46.242668 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:46.242628 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:46.295571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.295547 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:46.342821 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:46.342805 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-88.ec2.internal\" not found" Jul 22 12:40:46.396260 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.396246 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 12:40:46.396593 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.396355 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 12:40:46.396593 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.396374 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 22 12:40:46.426921 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.426901 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:46.430871 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.430854 2576 apiserver.go:52] "Watching apiserver" Jul 22 12:40:46.438326 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.438307 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 12:40:46.438665 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.438646 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-l9jtw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc","openshift-cluster-node-tuning-operator/tuned-k976g","openshift-image-registry/node-ca-7kfr8","openshift-multus/multus-ks584","openshift-network-diagnostics/network-check-target-6wx8p","openshift-network-operator/iptables-alerter-wc958","openshift-ovn-kubernetes/ovnkube-node-cdpvh","kube-system/global-pull-secret-syncer-4hvsk","openshift-dns/node-resolver-qh6q7","openshift-multus/multus-additional-cni-plugins-zvfg7","openshift-multus/network-metrics-daemon-tn8h2"] Jul 22 12:40:46.439963 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.439947 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.441914 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.441895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:46.442451 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.442429 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cfklb\"" Jul 22 12:40:46.442622 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.442593 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:40:46.442689 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.442659 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 12:40:46.443808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.443346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ks584" Jul 22 12:40:46.444528 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.444503 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 12:40:46.444966 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.444948 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 12:40:46.445098 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.445079 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlg9p\"" Jul 22 12:40:46.445622 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.445571 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 12:40:46.446521 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.445956 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:46.446521 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:46.446034 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:40:46.446521 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.446063 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6f6vp\"" Jul 22 12:40:46.446521 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.446161 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 12:40:46.446521 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.446481 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 12:40:46.446801 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.446780 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 12:40:46.447043 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.447028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:46.448062 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.448033 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 12:40:46.449032 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.449020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.449770 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.449756 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:40:46.450425 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.450411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.450494 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.450449 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9gsbd\"" Jul 22 12:40:46.450494 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.450464 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 12:40:46.450591 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.450506 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 12:40:46.451387 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.451372 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 12:40:46.451496 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.451406 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v8zsk\"" Jul 22 12:40:46.451496 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.451425 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 12:40:46.451496 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.451408 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 12:40:46.451657 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.451512 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 12:40:46.451707 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.451672 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:46.451707 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.451676 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 12:40:46.451707 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.451696 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 12:40:46.451837 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:46.451735 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:40:46.452392 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.452378 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" Jul 22 12:40:46.452458 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.452411 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 12:40:46.452657 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.452643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j962k\"" Jul 22 12:40:46.452816 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.452804 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 12:40:46.453622 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.453610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:40:46.453704 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.453691 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 12:40:46.455170 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.455157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:46.456280 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.456261 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pbtqc\"" Jul 22 12:40:46.456383 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.456368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:46.456465 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.456451 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 12:40:46.456517 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:46.456431 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:40:46.456570 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.456518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 12:40:46.457220 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-host\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:46.457311 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-cni-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.457311 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-cnibin\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.457311 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.457437 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-slash\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.457437 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-var-lib-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.457437 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457360 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysconfig\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.457437 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-cni-bin\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.457437 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8f8c309-00be-466a-94db-f8ce110a007f-ovn-node-metrics-cert\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-run-netns\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457546 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-cni-multus\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-daemon-config\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457585 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457622 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-kubernetes\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.457641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457639 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457651 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-socket-dir-parent\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbbj\" (UniqueName: \"kubernetes.io/projected/d8f286d0-2bca-4211-9499-d80776bd4d35-kube-api-access-stbbj\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-systemd-units\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457753 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bkr29\"" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-cni-netd\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-env-overrides\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-kubelet\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457905 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysctl-d\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-systemd\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-systemd\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.457998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.458028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-ovn\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-node-log\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1347662-9683-4412-9e63-61017d9d3be3-agent-certs\") pod \"konnectivity-agent-l9jtw\" (UID: \"a1347662-9683-4412-9e63-61017d9d3be3\") " pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458108 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-tuned\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458140 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-tmp\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-hostroot\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1347662-9683-4412-9e63-61017d9d3be3-konnectivity-ca\") pod \"konnectivity-agent-l9jtw\" (UID: \"a1347662-9683-4412-9e63-61017d9d3be3\") " pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-run\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-etc-kubernetes\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-os-release\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwnb5\" (UniqueName: \"kubernetes.io/projected/adc3ea48-dff1-4bf1-8dfc-f134f768323b-kube-api-access-hwnb5\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99q8k\" (UniqueName: \"kubernetes.io/projected/117cd6df-bf75-45e5-a516-3b6ad13bd929-kube-api-access-99q8k\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-log-socket\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjr2q\" (UniqueName: \"kubernetes.io/projected/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-kube-api-access-zjr2q\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-os-release\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33d75c32-fa71-44b1-8ad9-c61fb486e685-cni-binary-copy\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-multus-certs\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.458558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-cni-bin\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-serviceca\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-netns\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-sys\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rc54\" (UniqueName: \"kubernetes.io/projected/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-kube-api-access-6rc54\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-system-cni-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-k8s-cni-cncf-io\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4dps\" (UniqueName: \"kubernetes.io/projected/33d75c32-fa71-44b1-8ad9-c61fb486e685-kube-api-access-r4dps\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-system-cni-dir\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458861 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458895 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-kubelet\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-etc-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-modprobe-d\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-host\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.459238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysctl-conf\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.458994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-var-lib-kubelet\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-conf-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cnibin\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/117cd6df-bf75-45e5-a516-3b6ad13bd929-iptables-alerter-script\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/117cd6df-bf75-45e5-a516-3b6ad13bd929-host-slash\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-ovnkube-config\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-ovnkube-script-lib\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjn6\" (UniqueName: \"kubernetes.io/projected/e8f8c309-00be-466a-94db-f8ce110a007f-kube-api-access-hwjn6\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:46.459694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.459252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-lib-modules\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:46.460997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.460982 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5qgh\"" Jul 22 12:40:46.462041 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.462026 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 12:40:46.462152 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.462139 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 12:40:46.465961 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.465947 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 12:40:46.471679 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.471665 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 12:40:46.471756 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.471746 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" Jul 22 12:40:46.471907 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.471894 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal"] Jul 22 12:40:46.479532 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.479519 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal"] Jul 22 12:40:46.480184 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.480173 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 12:40:46.494655 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.494618 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x8krg" Jul 22 12:40:46.504643 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.504619 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 12:35:46 +0000 UTC" deadline="2027-12-16 18:54:00.368823638 +0000 UTC" Jul 22 12:40:46.504643 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.504640 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12294h13m13.864185728s" Jul 22 12:40:46.506519 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:46.506500 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x8krg" Jul 22 12:40:47.656861 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.656846 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 12:40:47.663607 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663589 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-cni-bin\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.663659 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-serviceca\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:47.663659 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663626 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-netns\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.663659 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-sys\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.663659 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rc54\" (UniqueName: \"kubernetes.io/projected/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-kube-api-access-6rc54\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-system-cni-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-k8s-cni-cncf-io\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-cni-bin\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4dps\" (UniqueName: \"kubernetes.io/projected/33d75c32-fa71-44b1-8ad9-c61fb486e685-kube-api-access-r4dps\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663757 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-netns\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-k8s-cni-cncf-io\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-system-cni-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-system-cni-dir\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.663831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663757 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-sys\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-system-cni-dir\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-kubelet\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-etc-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-kubelet\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663965 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-etc-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.663969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-modprobe-d\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-host\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-sys-fs\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-modprobe-d\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysctl-conf\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-host\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-var-lib-kubelet\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-conf-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.664223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-serviceca\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cnibin\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/117cd6df-bf75-45e5-a516-3b6ad13bd929-iptables-alerter-script\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664220 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-conf-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysctl-conf\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-var-lib-kubelet\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/117cd6df-bf75-45e5-a516-3b6ad13bd929-host-slash\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-ovnkube-config\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cnibin\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/117cd6df-bf75-45e5-a516-3b6ad13bd929-host-slash\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.664369 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-ovnkube-script-lib\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.664439 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs podName:d8f286d0-2bca-4211-9499-d80776bd4d35 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:48.164418952 +0000 UTC m=+1.969135855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs") pod "network-metrics-daemon-tn8h2" (UID: "d8f286d0-2bca-4211-9499-d80776bd4d35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjn6\" (UniqueName: \"kubernetes.io/projected/e8f8c309-00be-466a-94db-f8ce110a007f-kube-api-access-hwjn6\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664485 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-lib-modules\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.664937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-host\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664571 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-lib-modules\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664604 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-host\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-cni-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-cnibin\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-ovnkube-config\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-cnibin\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-slash\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664798 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/117cd6df-bf75-45e5-a516-3b6ad13bd929-iptables-alerter-script\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664808 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-var-lib-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664820 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-cni-dir\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-ovnkube-script-lib\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysconfig\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664842 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-slash\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysconfig\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664867 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-var-lib-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.665731 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-cni-bin\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-etc-selinux\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-cni-bin\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.664962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-tmp-dir\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8f8c309-00be-466a-94db-f8ce110a007f-ovn-node-metrics-cert\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665116 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-device-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665136 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-run-netns\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-cni-multus\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-daemon-config\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-cni-binary-copy\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-run-netns\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665227 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665255 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-cni-multus\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e10dae84-5ee9-4c3c-a86c-608de7b8504e-dbus\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:47.666459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665289 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-kubernetes\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-socket-dir-parent\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbbj\" (UniqueName: \"kubernetes.io/projected/d8f286d0-2bca-4211-9499-d80776bd4d35-kube-api-access-stbbj\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-registration-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/adc3ea48-dff1-4bf1-8dfc-f134f768323b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-systemd-units\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665422 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-socket-dir-parent\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665427 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-cni-netd\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-systemd-units\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-env-overrides\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-kubernetes\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-cni-netd\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-kubelet\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-socket-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665518 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysctl-d\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-var-lib-kubelet\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.666998 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-systemd\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e10dae84-5ee9-4c3c-a86c-608de7b8504e-kubelet-config\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-systemd\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-systemd\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-sysctl-d\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-systemd\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-openvswitch\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-ovn\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-node-log\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e8f8c309-00be-466a-94db-f8ce110a007f-env-overrides\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33d75c32-fa71-44b1-8ad9-c61fb486e685-multus-daemon-config\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1347662-9683-4412-9e63-61017d9d3be3-agent-certs\") pod \"konnectivity-agent-l9jtw\" (UID: \"a1347662-9683-4412-9e63-61017d9d3be3\") " pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665792 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-run-ovn\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-tuned\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665816 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-node-log\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-tmp\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-hostroot\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.667522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1347662-9683-4412-9e63-61017d9d3be3-konnectivity-ca\") pod \"konnectivity-agent-l9jtw\" (UID: \"a1347662-9683-4412-9e63-61017d9d3be3\") " pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-run\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-etc-kubernetes\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-os-release\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-run\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.665991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/adc3ea48-dff1-4bf1-8dfc-f134f768323b-os-release\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-etc-kubernetes\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwnb5\" (UniqueName: \"kubernetes.io/projected/adc3ea48-dff1-4bf1-8dfc-f134f768323b-kube-api-access-hwnb5\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99q8k\" (UniqueName: \"kubernetes.io/projected/117cd6df-bf75-45e5-a516-3b6ad13bd929-kube-api-access-99q8k\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666124 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8sn\" (UniqueName: \"kubernetes.io/projected/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-kube-api-access-2k8sn\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666171 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-log-socket\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjr2q\" (UniqueName: \"kubernetes.io/projected/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-kube-api-access-zjr2q\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-hostroot\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-os-release\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33d75c32-fa71-44b1-8ad9-c61fb486e685-cni-binary-copy\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-multus-certs\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.667975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99j9d\" (UniqueName: \"kubernetes.io/projected/bd070942-1c2e-487b-b373-4bd8455aea83-kube-api-access-99j9d\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-hosts-file\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666596 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-host-run-ovn-kubernetes\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666666 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-host-run-multus-certs\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666718 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e8f8c309-00be-466a-94db-f8ce110a007f-log-socket\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666859 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a1347662-9683-4412-9e63-61017d9d3be3-konnectivity-ca\") pod \"konnectivity-agent-l9jtw\" (UID: \"a1347662-9683-4412-9e63-61017d9d3be3\") " pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666942 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33d75c32-fa71-44b1-8ad9-c61fb486e685-os-release\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.666986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33d75c32-fa71-44b1-8ad9-c61fb486e685-cni-binary-copy\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.668315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-etc-tuned\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.668368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-tmp\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.668473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8f8c309-00be-466a-94db-f8ce110a007f-ovn-node-metrics-cert\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.668571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.668561 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a1347662-9683-4412-9e63-61017d9d3be3-agent-certs\") pod \"konnectivity-agent-l9jtw\" (UID: \"a1347662-9683-4412-9e63-61017d9d3be3\") " pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:40:47.672761 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.672738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4dps\" (UniqueName: \"kubernetes.io/projected/33d75c32-fa71-44b1-8ad9-c61fb486e685-kube-api-access-r4dps\") pod \"multus-ks584\" (UID: \"33d75c32-fa71-44b1-8ad9-c61fb486e685\") " pod="openshift-multus/multus-ks584" Jul 22 12:40:47.672831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.672815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rc54\" (UniqueName: \"kubernetes.io/projected/cdab67a7-9489-4d5c-ad85-098e8a5c7c7d-kube-api-access-6rc54\") pod \"node-ca-7kfr8\" (UID: \"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d\") " pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:47.673752 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.673736 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:47.673798 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.673757 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:47.673798 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.673771 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zh46m for pod openshift-network-diagnostics/network-check-target-6wx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:47.673862 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.673838 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m podName:ceea1b5d-f802-46e1-8391-111cfa88ed6d nodeName:}" failed. No retries permitted until 2026-07-22 12:40:48.173820452 +0000 UTC m=+1.978537350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zh46m" (UniqueName: "kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m") pod "network-check-target-6wx8p" (UID: "ceea1b5d-f802-46e1-8391-111cfa88ed6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:47.676284 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.676158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwnb5\" (UniqueName: \"kubernetes.io/projected/adc3ea48-dff1-4bf1-8dfc-f134f768323b-kube-api-access-hwnb5\") pod \"multus-additional-cni-plugins-zvfg7\" (UID: \"adc3ea48-dff1-4bf1-8dfc-f134f768323b\") " pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.676284 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.676257 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjn6\" (UniqueName: \"kubernetes.io/projected/e8f8c309-00be-466a-94db-f8ce110a007f-kube-api-access-hwjn6\") pod \"ovnkube-node-cdpvh\" (UID: \"e8f8c309-00be-466a-94db-f8ce110a007f\") " pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.676746 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.676729 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbbj\" (UniqueName: \"kubernetes.io/projected/d8f286d0-2bca-4211-9499-d80776bd4d35-kube-api-access-stbbj\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:47.676816 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.676790 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99q8k\" (UniqueName: \"kubernetes.io/projected/117cd6df-bf75-45e5-a516-3b6ad13bd929-kube-api-access-99q8k\") pod \"iptables-alerter-wc958\" (UID: \"117cd6df-bf75-45e5-a516-3b6ad13bd929\") " pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:47.676990 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.676969 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjr2q\" (UniqueName: \"kubernetes.io/projected/a27aebc4-4a9c-496e-a8a5-0c7caaa8b880-kube-api-access-zjr2q\") pod \"tuned-k976g\" (UID: \"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880\") " pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.766806 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-sys-fs\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-etc-selinux\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-tmp-dir\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-device-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766904 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-sys-fs\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e10dae84-5ee9-4c3c-a86c-608de7b8504e-dbus\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-registration-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-socket-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.766995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e10dae84-5ee9-4c3c-a86c-608de7b8504e-kubelet-config\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-etc-selinux\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:47.767069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8sn\" (UniqueName: \"kubernetes.io/projected/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-kube-api-access-2k8sn\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99j9d\" (UniqueName: \"kubernetes.io/projected/bd070942-1c2e-487b-b373-4bd8455aea83-kube-api-access-99j9d\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-registration-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-hosts-file\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767185 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-hosts-file\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-socket-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e10dae84-5ee9-4c3c-a86c-608de7b8504e-kubelet-config\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd070942-1c2e-487b-b373-4bd8455aea83-device-dir\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-tmp-dir\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.767420 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e10dae84-5ee9-4c3c-a86c-608de7b8504e-dbus\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.767462 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:47.767606 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.767521 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret podName:e10dae84-5ee9-4c3c-a86c-608de7b8504e nodeName:}" failed. No retries permitted until 2026-07-22 12:40:48.267495122 +0000 UTC m=+2.072211996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret") pod "global-pull-secret-syncer-4hvsk" (UID: "e10dae84-5ee9-4c3c-a86c-608de7b8504e") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:47.776189 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.776162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8sn\" (UniqueName: \"kubernetes.io/projected/713f7499-9b3a-4f9c-b6a8-8682fb8f507d-kube-api-access-2k8sn\") pod \"node-resolver-qh6q7\" (UID: \"713f7499-9b3a-4f9c-b6a8-8682fb8f507d\") " pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.776477 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.776461 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99j9d\" (UniqueName: \"kubernetes.io/projected/bd070942-1c2e-487b-b373-4bd8455aea83-kube-api-access-99j9d\") pod \"aws-ebs-csi-driver-node-t5sdc\" (UID: \"bd070942-1c2e-487b-b373-4bd8455aea83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.842740 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.842706 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349801908a8762aee344d69fa297f1e3.slice/crio-52b3866ea1b035790081957af187a00b110026d4125cc25ea135909def3c7c8e WatchSource:0}: Error finding container 52b3866ea1b035790081957af187a00b110026d4125cc25ea135909def3c7c8e: Status 404 returned error can't find the container with id 52b3866ea1b035790081957af187a00b110026d4125cc25ea135909def3c7c8e Jul 22 12:40:47.842966 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.842941 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69c0cdb4de82ba0de3b9240fc59ceea.slice/crio-d3e97b201ea06d750c43b42ecaf6cfeee07c0083165188a8e3b27302a282edec WatchSource:0}: Error finding container d3e97b201ea06d750c43b42ecaf6cfeee07c0083165188a8e3b27302a282edec: Status 404 returned error can't find the container with id d3e97b201ea06d750c43b42ecaf6cfeee07c0083165188a8e3b27302a282edec Jul 22 12:40:47.845690 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.845675 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:40:47.869741 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.869692 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k976g" Jul 22 12:40:47.876278 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.876262 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27aebc4_4a9c_496e_a8a5_0c7caaa8b880.slice/crio-c02261b2852c7fa7537ef71b6c465f9037080957e00c8301d4c35deac60b983a WatchSource:0}: Error finding container c02261b2852c7fa7537ef71b6c465f9037080957e00c8301d4c35deac60b983a: Status 404 returned error can't find the container with id c02261b2852c7fa7537ef71b6c465f9037080957e00c8301d4c35deac60b983a Jul 22 12:40:47.876586 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.876571 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:47.882768 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.882751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7kfr8" Jul 22 12:40:47.890305 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.890283 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdab67a7_9489_4d5c_ad85_098e8a5c7c7d.slice/crio-8dc9119799b89de62d324798bcee72c09b074fc06b29bf956cc067ef94aa12a0 WatchSource:0}: Error finding container 8dc9119799b89de62d324798bcee72c09b074fc06b29bf956cc067ef94aa12a0: Status 404 returned error can't find the container with id 8dc9119799b89de62d324798bcee72c09b074fc06b29bf956cc067ef94aa12a0 Jul 22 12:40:47.896851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.896834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ks584" Jul 22 12:40:47.904212 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.904194 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d75c32_fa71_44b1_8ad9_c61fb486e685.slice/crio-c4181cb519e2ebd78d0e90c68dc786c9dbf70cefafd499e1966b4a35fc278129 WatchSource:0}: Error finding container c4181cb519e2ebd78d0e90c68dc786c9dbf70cefafd499e1966b4a35fc278129: Status 404 returned error can't find the container with id c4181cb519e2ebd78d0e90c68dc786c9dbf70cefafd499e1966b4a35fc278129 Jul 22 12:40:47.908845 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.908830 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wc958" Jul 22 12:40:47.915454 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.915436 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117cd6df_bf75_45e5_a516_3b6ad13bd929.slice/crio-30abfa8244471e84111e72665f5887eb0832fe7157857be9ff73ddc12470f5e7 WatchSource:0}: Error finding container 30abfa8244471e84111e72665f5887eb0832fe7157857be9ff73ddc12470f5e7: Status 404 returned error can't find the container with id 30abfa8244471e84111e72665f5887eb0832fe7157857be9ff73ddc12470f5e7 Jul 22 12:40:47.928087 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.928068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:40:47.934488 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.934472 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f8c309_00be_466a_94db_f8ce110a007f.slice/crio-0a9f39c511970629c511e9120ad07841067aab8d61890d7d90261866cf36ab8e WatchSource:0}: Error finding container 0a9f39c511970629c511e9120ad07841067aab8d61890d7d90261866cf36ab8e: Status 404 returned error can't find the container with id 0a9f39c511970629c511e9120ad07841067aab8d61890d7d90261866cf36ab8e Jul 22 12:40:47.939455 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.939442 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" Jul 22 12:40:47.946137 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.946116 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc3ea48_dff1_4bf1_8dfc_f134f768323b.slice/crio-d8006e1e70d8f12368e8e18e6e199317dfba139cdece917abf9835bf14e7e12f WatchSource:0}: Error finding container d8006e1e70d8f12368e8e18e6e199317dfba139cdece917abf9835bf14e7e12f: Status 404 returned error can't find the container with id d8006e1e70d8f12368e8e18e6e199317dfba139cdece917abf9835bf14e7e12f Jul 22 12:40:47.953901 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.953886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:40:47.961146 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.961126 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1347662_9683_4412_9e63_61017d9d3be3.slice/crio-94687e99187ce75de773810f4d178915d7272721537dab758385640274c87705 WatchSource:0}: Error finding container 94687e99187ce75de773810f4d178915d7272721537dab758385640274c87705: Status 404 returned error can't find the container with id 94687e99187ce75de773810f4d178915d7272721537dab758385640274c87705 Jul 22 12:40:47.983739 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.983721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" Jul 22 12:40:47.987489 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:47.987475 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qh6q7" Jul 22 12:40:47.989882 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.989863 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd070942_1c2e_487b_b373_4bd8455aea83.slice/crio-48a838d4095134cfbb96578b316b4a04a1304903359b95cccc98998f7eaf3a14 WatchSource:0}: Error finding container 48a838d4095134cfbb96578b316b4a04a1304903359b95cccc98998f7eaf3a14: Status 404 returned error can't find the container with id 48a838d4095134cfbb96578b316b4a04a1304903359b95cccc98998f7eaf3a14 Jul 22 12:40:47.994122 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:40:47.994103 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713f7499_9b3a_4f9c_b6a8_8682fb8f507d.slice/crio-8db76461fab7064fec5e793a94bff05fd40a797c09f0d3a47a3a90cccfa8694d WatchSource:0}: Error finding container 8db76461fab7064fec5e793a94bff05fd40a797c09f0d3a47a3a90cccfa8694d: Status 404 returned error can't find the container with id 8db76461fab7064fec5e793a94bff05fd40a797c09f0d3a47a3a90cccfa8694d Jul 22 12:40:47.995506 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.995488 2576 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-qh6q7_openshift-dns(713f7499-9b3a-4f9c-b6a8-8682fb8f507d): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 22 12:40:47.995552 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:47.995524 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-qh6q7" podUID="713f7499-9b3a-4f9c-b6a8-8682fb8f507d" Jul 22 12:40:48.170688 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.170603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:48.170791 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.170729 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:48.170791 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.170775 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs podName:d8f286d0-2bca-4211-9499-d80776bd4d35 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:49.170762312 +0000 UTC m=+2.975479188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs") pod "network-metrics-daemon-tn8h2" (UID: "d8f286d0-2bca-4211-9499-d80776bd4d35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:48.271844 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.271814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:48.271987 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.271866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:48.271987 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.271947 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:48.271987 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.271962 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:48.271987 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.271975 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:48.271987 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.271984 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zh46m for pod openshift-network-diagnostics/network-check-target-6wx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:48.272162 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.272008 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret podName:e10dae84-5ee9-4c3c-a86c-608de7b8504e nodeName:}" failed. No retries permitted until 2026-07-22 12:40:49.271989945 +0000 UTC m=+3.076706835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret") pod "global-pull-secret-syncer-4hvsk" (UID: "e10dae84-5ee9-4c3c-a86c-608de7b8504e") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:48.272162 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.272039 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m podName:ceea1b5d-f802-46e1-8391-111cfa88ed6d nodeName:}" failed. No retries permitted until 2026-07-22 12:40:49.272029249 +0000 UTC m=+3.076746123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh46m" (UniqueName: "kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m") pod "network-check-target-6wx8p" (UID: "ceea1b5d-f802-46e1-8391-111cfa88ed6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:48.406281 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.406251 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:48.613262 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.613096 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 12:35:47 +0000 UTC" deadline="2028-03-30 23:56:46.557780234 +0000 UTC" Jul 22 12:40:48.613262 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.613136 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14819h15m57.94464957s" Jul 22 12:40:48.721347 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.721278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k976g" event={"ID":"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880","Type":"ContainerStarted","Data":"c02261b2852c7fa7537ef71b6c465f9037080957e00c8301d4c35deac60b983a"} Jul 22 12:40:48.726612 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.726503 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l9jtw" event={"ID":"a1347662-9683-4412-9e63-61017d9d3be3","Type":"ContainerStarted","Data":"94687e99187ce75de773810f4d178915d7272721537dab758385640274c87705"} Jul 22 12:40:48.731345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.731318 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wc958" event={"ID":"117cd6df-bf75-45e5-a516-3b6ad13bd929","Type":"ContainerStarted","Data":"30abfa8244471e84111e72665f5887eb0832fe7157857be9ff73ddc12470f5e7"} Jul 22 12:40:48.747500 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.747467 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ks584" event={"ID":"33d75c32-fa71-44b1-8ad9-c61fb486e685","Type":"ContainerStarted","Data":"c4181cb519e2ebd78d0e90c68dc786c9dbf70cefafd499e1966b4a35fc278129"} Jul 22 12:40:48.755753 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.755720 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" event={"ID":"c69c0cdb4de82ba0de3b9240fc59ceea","Type":"ContainerStarted","Data":"d3e97b201ea06d750c43b42ecaf6cfeee07c0083165188a8e3b27302a282edec"} Jul 22 12:40:48.761065 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.761025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" event={"ID":"349801908a8762aee344d69fa297f1e3","Type":"ContainerStarted","Data":"52b3866ea1b035790081957af187a00b110026d4125cc25ea135909def3c7c8e"} Jul 22 12:40:48.771870 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.771841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qh6q7" event={"ID":"713f7499-9b3a-4f9c-b6a8-8682fb8f507d","Type":"ContainerStarted","Data":"8db76461fab7064fec5e793a94bff05fd40a797c09f0d3a47a3a90cccfa8694d"} Jul 22 12:40:48.774112 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:48.774046 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e83687a3e51e78c37199d4d4504faf8bfeec5d995d123db950ef57fe67957539\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-qh6q7" podUID="713f7499-9b3a-4f9c-b6a8-8682fb8f507d" Jul 22 12:40:48.776296 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.776271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" event={"ID":"bd070942-1c2e-487b-b373-4bd8455aea83","Type":"ContainerStarted","Data":"48a838d4095134cfbb96578b316b4a04a1304903359b95cccc98998f7eaf3a14"} Jul 22 12:40:48.789677 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.789636 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" event={"ID":"adc3ea48-dff1-4bf1-8dfc-f134f768323b","Type":"ContainerStarted","Data":"d8006e1e70d8f12368e8e18e6e199317dfba139cdece917abf9835bf14e7e12f"} Jul 22 12:40:48.804446 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.804420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"0a9f39c511970629c511e9120ad07841067aab8d61890d7d90261866cf36ab8e"} Jul 22 12:40:48.821373 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:48.821343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7kfr8" event={"ID":"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d","Type":"ContainerStarted","Data":"8dc9119799b89de62d324798bcee72c09b074fc06b29bf956cc067ef94aa12a0"} Jul 22 12:40:49.179219 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:49.179178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:49.179391 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.179366 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:49.179452 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.179430 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs podName:d8f286d0-2bca-4211-9499-d80776bd4d35 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:51.179411004 +0000 UTC m=+4.984127895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs") pod "network-metrics-daemon-tn8h2" (UID: "d8f286d0-2bca-4211-9499-d80776bd4d35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:49.280703 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:49.280619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:49.280855 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:49.280728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:49.280922 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.280880 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:49.280922 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.280899 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:49.280922 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.280912 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zh46m for pod openshift-network-diagnostics/network-check-target-6wx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:49.281100 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.280969 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m podName:ceea1b5d-f802-46e1-8391-111cfa88ed6d nodeName:}" failed. No retries permitted until 2026-07-22 12:40:51.280950777 +0000 UTC m=+5.085667667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh46m" (UniqueName: "kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m") pod "network-check-target-6wx8p" (UID: "ceea1b5d-f802-46e1-8391-111cfa88ed6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:49.281399 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.281380 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:49.281497 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.281435 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret podName:e10dae84-5ee9-4c3c-a86c-608de7b8504e nodeName:}" failed. No retries permitted until 2026-07-22 12:40:51.281420289 +0000 UTC m=+5.086137163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret") pod "global-pull-secret-syncer-4hvsk" (UID: "e10dae84-5ee9-4c3c-a86c-608de7b8504e") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:49.613810 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:49.613702 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 12:35:47 +0000 UTC" deadline="2028-04-26 23:07:33.428652912 +0000 UTC" Jul 22 12:40:49.613810 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:49.613762 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15466h26m43.814895116s" Jul 22 12:40:49.708434 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:49.707450 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:49.708434 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.707589 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:40:49.708434 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:49.708134 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:49.708434 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.708232 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:40:49.708434 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:49.708309 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:49.708434 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:49.708390 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:40:50.015829 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:50.015492 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:51.196999 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:51.196947 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:51.197474 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.197149 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:51.197474 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.197213 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs podName:d8f286d0-2bca-4211-9499-d80776bd4d35 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:55.197195256 +0000 UTC m=+9.001912154 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs") pod "network-metrics-daemon-tn8h2" (UID: "d8f286d0-2bca-4211-9499-d80776bd4d35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:51.297573 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:51.297536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:51.297743 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:51.297599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:51.297743 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.297718 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:51.297844 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.297783 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret podName:e10dae84-5ee9-4c3c-a86c-608de7b8504e nodeName:}" failed. No retries permitted until 2026-07-22 12:40:55.297765103 +0000 UTC m=+9.102481979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret") pod "global-pull-secret-syncer-4hvsk" (UID: "e10dae84-5ee9-4c3c-a86c-608de7b8504e") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:51.297908 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.297877 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:51.297908 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.297891 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:51.297908 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.297903 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zh46m for pod openshift-network-diagnostics/network-check-target-6wx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:51.298065 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.297937 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m podName:ceea1b5d-f802-46e1-8391-111cfa88ed6d nodeName:}" failed. No retries permitted until 2026-07-22 12:40:55.297926292 +0000 UTC m=+9.102643169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh46m" (UniqueName: "kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m") pod "network-check-target-6wx8p" (UID: "ceea1b5d-f802-46e1-8391-111cfa88ed6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:51.707103 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:51.707068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:51.707268 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:51.707108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:51.707268 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.707203 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:40:51.707361 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.707276 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:40:51.707361 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:51.707329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:51.707442 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:51.707397 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:40:53.706824 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:53.706789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:53.707379 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:53.706908 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:40:53.707379 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:53.707331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:53.707497 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:53.707418 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:40:53.707497 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:53.707485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:53.707596 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:53.707567 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:40:55.233139 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:55.233107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:55.233523 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.233239 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:55.233523 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.233292 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs podName:d8f286d0-2bca-4211-9499-d80776bd4d35 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:03.233279594 +0000 UTC m=+17.037996468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs") pod "network-metrics-daemon-tn8h2" (UID: "d8f286d0-2bca-4211-9499-d80776bd4d35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:55.334402 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:55.334084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:55.334402 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:55.334179 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:55.334402 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.334226 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:55.334402 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.334283 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:55.334402 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.334303 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:55.334402 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.334317 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zh46m for pod openshift-network-diagnostics/network-check-target-6wx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:55.334402 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.334284 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret podName:e10dae84-5ee9-4c3c-a86c-608de7b8504e nodeName:}" failed. No retries permitted until 2026-07-22 12:41:03.334270267 +0000 UTC m=+17.138987141 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret") pod "global-pull-secret-syncer-4hvsk" (UID: "e10dae84-5ee9-4c3c-a86c-608de7b8504e") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:55.334402 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.334376 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m podName:ceea1b5d-f802-46e1-8391-111cfa88ed6d nodeName:}" failed. No retries permitted until 2026-07-22 12:41:03.334360884 +0000 UTC m=+17.139077765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh46m" (UniqueName: "kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m") pod "network-check-target-6wx8p" (UID: "ceea1b5d-f802-46e1-8391-111cfa88ed6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:55.708377 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:55.707609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:55.708377 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.707729 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:40:55.708377 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:55.708122 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:55.708377 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.708228 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:40:55.708377 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:55.708274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:55.708377 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:55.708339 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:40:57.707415 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:57.707381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:57.707415 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:57.707417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:57.707902 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:57.707381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:57.707902 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:57.707496 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:40:57.707902 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:57.707561 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:40:57.707902 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:57.707639 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:40:59.707279 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:59.707218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:40:59.707631 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:59.707333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:40:59.707631 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:40:59.707346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:40:59.707631 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:59.707343 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:40:59.707631 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:59.707432 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:40:59.707806 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:40:59.707691 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:01.707575 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:01.707541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:01.707997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:01.707541 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:01.707997 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:01.707657 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:01.707997 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:01.707674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:01.707997 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:01.707769 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:01.707997 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:01.707862 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:03.299759 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:03.299730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:03.300125 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.299843 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:03.300125 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.299914 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs podName:d8f286d0-2bca-4211-9499-d80776bd4d35 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:19.299892951 +0000 UTC m=+33.104609830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs") pod "network-metrics-daemon-tn8h2" (UID: "d8f286d0-2bca-4211-9499-d80776bd4d35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:03.400912 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:03.400878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:03.401087 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:03.400942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:03.401087 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.401044 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:03.401087 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.401067 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:03.401087 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.401080 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zh46m for pod openshift-network-diagnostics/network-check-target-6wx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:03.401258 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.401088 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:03.401258 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.401142 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m podName:ceea1b5d-f802-46e1-8391-111cfa88ed6d nodeName:}" failed. No retries permitted until 2026-07-22 12:41:19.401123748 +0000 UTC m=+33.205840636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh46m" (UniqueName: "kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m") pod "network-check-target-6wx8p" (UID: "ceea1b5d-f802-46e1-8391-111cfa88ed6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:03.401258 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.401162 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret podName:e10dae84-5ee9-4c3c-a86c-608de7b8504e nodeName:}" failed. No retries permitted until 2026-07-22 12:41:19.40115244 +0000 UTC m=+33.205869320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret") pod "global-pull-secret-syncer-4hvsk" (UID: "e10dae84-5ee9-4c3c-a86c-608de7b8504e") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:03.707215 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:03.707188 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:03.707410 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:03.707290 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:03.707410 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.707290 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:03.707410 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:03.707385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:03.707410 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.707390 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:03.707573 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:03.707470 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:05.707276 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:05.707239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:05.707276 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:05.707276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:05.707738 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:05.707360 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:05.707738 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:05.707416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:05.707738 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:05.707534 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:05.707738 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:05.707625 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:07.707695 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:07.707663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:07.707695 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:07.707697 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:07.708125 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:07.707771 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:07.708125 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:07.707771 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:07.708125 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:07.707886 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:07.708125 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:07.707981 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:08.863300 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.862892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7kfr8" event={"ID":"cdab67a7-9489-4d5c-ad85-098e8a5c7c7d","Type":"ContainerStarted","Data":"ab4097c5477b319b87fdf4fd598730b00f07a423b057e33c5d8d4a68b2f0e823"} Jul 22 12:41:08.864673 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.864652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k976g" event={"ID":"a27aebc4-4a9c-496e-a8a5-0c7caaa8b880","Type":"ContainerStarted","Data":"cada291d8da436550aea57b914e7a536e963c978feabc8227a5e5511281a18e0"} Jul 22 12:41:08.866007 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.865986 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l9jtw" event={"ID":"a1347662-9683-4412-9e63-61017d9d3be3","Type":"ContainerStarted","Data":"75b3ce82458dd1b28f8d1952a98a94efdfab2d9a971d25598488c57b792862d3"} Jul 22 12:41:08.867417 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.867394 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ks584" event={"ID":"33d75c32-fa71-44b1-8ad9-c61fb486e685","Type":"ContainerStarted","Data":"0038e0abd238ce6a7931c056f5fd4cdc4111ca28653f044d9dce03e3c5fc1970"} Jul 22 12:41:08.868860 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.868842 2576 generic.go:356] "Generic (PLEG): container finished" podID="c69c0cdb4de82ba0de3b9240fc59ceea" containerID="fd8b734a8119a93e402830ec8de2252a3512bc2d3190c23dcf26ddfee5cb592a" exitCode=0 Jul 22 12:41:08.868934 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.868900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" event={"ID":"c69c0cdb4de82ba0de3b9240fc59ceea","Type":"ContainerDied","Data":"fd8b734a8119a93e402830ec8de2252a3512bc2d3190c23dcf26ddfee5cb592a"} Jul 22 12:41:08.870314 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.870290 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" event={"ID":"349801908a8762aee344d69fa297f1e3","Type":"ContainerStarted","Data":"616c91ab6f503959f704cf4691ba7bb441eae265dfcd1986204d4120423a7620"} Jul 22 12:41:08.871560 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.871535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qh6q7" event={"ID":"713f7499-9b3a-4f9c-b6a8-8682fb8f507d","Type":"ContainerStarted","Data":"47afdff4e370192ab6ba2689be4de0728c9c4452378e783c9fb6d21fa0b2d6da"} Jul 22 12:41:08.872758 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.872741 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" event={"ID":"bd070942-1c2e-487b-b373-4bd8455aea83","Type":"ContainerStarted","Data":"6668925d68c5da7ddabbd3db2166e688717508f286b94b95c8c4a8b9b7a037f7"} Jul 22 12:41:08.874083 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.874056 2576 generic.go:356] "Generic (PLEG): container finished" podID="adc3ea48-dff1-4bf1-8dfc-f134f768323b" containerID="878a289a0318477b8b08b83e46c48ec80096a77d799922a509c57ba01310f32e" exitCode=0 Jul 22 12:41:08.874163 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.874141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" event={"ID":"adc3ea48-dff1-4bf1-8dfc-f134f768323b","Type":"ContainerDied","Data":"878a289a0318477b8b08b83e46c48ec80096a77d799922a509c57ba01310f32e"} Jul 22 12:41:08.876879 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.876860 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:41:08.877283 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.877265 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8f8c309-00be-466a-94db-f8ce110a007f" containerID="f4b5ee189eb0c8694196803cf79be1675f26668a3949619b3e3ca144d7cd4fab" exitCode=1 Jul 22 12:41:08.877367 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.877291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"59c8cdd649fd619e8fefe4fe3802ec07985a9756b1b5d7560934b0622d6e0cf3"} Jul 22 12:41:08.877367 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.877335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"a3b3645f6f959a449d1befcf93fb4c79f84427185cf2053ffd34cf928a472bd7"} Jul 22 12:41:08.877431 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.877368 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"26ee9867d9d121580fa73a93bf6245bbcd11e7ecfab29c9dcacbe0f095532d6f"} Jul 22 12:41:08.877431 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.877395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"6eca7c201accf44b9249f1bf19464c3ac4c6526c867c9450fe9360411627c7d9"} Jul 22 12:41:08.877431 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.877420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerDied","Data":"f4b5ee189eb0c8694196803cf79be1675f26668a3949619b3e3ca144d7cd4fab"} Jul 22 12:41:08.877518 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.877448 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"bd93c686a4a224f2bb34d324e9bf207275ed38be23a95a22bf16357806d3cb0c"} Jul 22 12:41:08.879790 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.879754 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7kfr8" podStartSLOduration=2.946214639 podStartE2EDuration="22.879738225s" totalImagesPullingTime="19.933523586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:08.879393416 +0000 UTC m=+22.684110311" watchObservedRunningTime="2026-07-22 12:41:08.879738225 +0000 UTC m=+22.684455121" Jul 22 12:41:08.899007 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.898973 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ks584" podStartSLOduration=2.977113013 podStartE2EDuration="22.898956281s" totalImagesPullingTime="19.921843268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:08.898640853 +0000 UTC m=+22.703357746" watchObservedRunningTime="2026-07-22 12:41:08.898956281 +0000 UTC m=+22.703673176" Jul 22 12:41:08.924289 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.924247 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l9jtw" podStartSLOduration=2.06053753 podStartE2EDuration="21.924233141s" totalImagesPullingTime="19.863695611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:08.923821302 +0000 UTC m=+22.728538197" watchObservedRunningTime="2026-07-22 12:41:08.924233141 +0000 UTC m=+22.728950036" Jul 22 12:41:08.954842 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.954804 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-88.ec2.internal" podStartSLOduration=21.954790416 podStartE2EDuration="21.954790416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:08.954399906 +0000 UTC m=+22.759116802" watchObservedRunningTime="2026-07-22 12:41:08.954790416 +0000 UTC m=+22.759507311" Jul 22 12:41:08.955289 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:08.955265 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qh6q7" podStartSLOduration=21.955253576 podStartE2EDuration="21.955253576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-22 12:41:08.940437861 +0000 UTC m=+22.745154759" watchObservedRunningTime="2026-07-22 12:41:08.955253576 +0000 UTC m=+22.759970473" Jul 22 12:41:09.044595 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:09.044555 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k976g" podStartSLOduration=3.09418102 podStartE2EDuration="23.04454113s" totalImagesPullingTime="19.95036011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:09.044101218 +0000 UTC m=+22.848818112" watchObservedRunningTime="2026-07-22 12:41:09.04454113 +0000 UTC m=+22.849258026" Jul 22 12:41:09.707340 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:09.707306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:09.707576 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:09.707346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:09.707576 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:09.707425 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:09.707576 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:09.707481 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:09.707576 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:09.707503 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:09.707768 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:09.707618 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:09.880502 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:09.880469 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wc958" event={"ID":"117cd6df-bf75-45e5-a516-3b6ad13bd929","Type":"ContainerStarted","Data":"47c758d1a39cec385b11fece7c9cd3975c8c580beef9c14805cf1ee7a48ea235"} Jul 22 12:41:09.900441 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:09.900333 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wc958" podStartSLOduration=3.991145764 podStartE2EDuration="23.900317861s" totalImagesPullingTime="19.909172097s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:09.899937798 +0000 UTC m=+23.704654694" watchObservedRunningTime="2026-07-22 12:41:09.900317861 +0000 UTC m=+23.705034756" Jul 22 12:41:10.153435 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.153172 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 12:41:10.630743 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.630645 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T12:41:10.153435425Z","UUID":"908485ab-2595-44fd-a38d-8ff3cd31d149","Handler":null,"Name":"","Endpoint":""} Jul 22 12:41:10.632569 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.632548 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 12:41:10.632696 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.632591 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 12:41:10.654307 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.654287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:41:10.654855 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.654836 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:41:10.885595 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.885527 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" event={"ID":"c69c0cdb4de82ba0de3b9240fc59ceea","Type":"ContainerStarted","Data":"7f234f3868025a9ca05ebe83aa242c85001b22b5df474dbaca12befa08590762"} Jul 22 12:41:10.888135 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.888106 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" event={"ID":"bd070942-1c2e-487b-b373-4bd8455aea83","Type":"ContainerStarted","Data":"f2defb15becd59fa5a641cd7cfc1d0037c7d2a1ec211258ede9634b306fbbbec"} Jul 22 12:41:10.902826 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:10.902787 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-88.ec2.internal" podStartSLOduration=23.902772809 podStartE2EDuration="23.902772809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:10.902264366 +0000 UTC m=+24.706981261" watchObservedRunningTime="2026-07-22 12:41:10.902772809 +0000 UTC m=+24.707489704" Jul 22 12:41:11.458487 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.458461 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:41:11.459026 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.458996 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l9jtw" Jul 22 12:41:11.707757 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.707675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:11.707757 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.707709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:11.707757 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.707709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:11.708005 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:11.707791 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:11.708005 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:11.707888 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:11.708005 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:11.707949 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:11.892558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.892520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" event={"ID":"bd070942-1c2e-487b-b373-4bd8455aea83","Type":"ContainerStarted","Data":"d84f83247d26ed43502ef96aebf991a8c932b71b0040f14adcccfc43fbda82f6"} Jul 22 12:41:11.895914 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.895895 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:41:11.896353 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.896322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"43c23aa11127cb27ba9e985e213a5205d015b7afe3a2b4e1b94e7760529ef382"} Jul 22 12:41:11.913204 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:11.913161 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t5sdc" podStartSLOduration=2.534842665 podStartE2EDuration="24.913145698s" totalImagesPullingTime="22.378303033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:11.912592963 +0000 UTC m=+25.717309859" watchObservedRunningTime="2026-07-22 12:41:11.913145698 +0000 UTC m=+25.717862593" Jul 22 12:41:13.707523 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.707365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:13.707918 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.707366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:13.707918 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:13.707591 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:13.707918 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:13.707659 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:13.707918 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.707396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:13.707918 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:13.707781 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:13.902654 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.902583 2576 generic.go:356] "Generic (PLEG): container finished" podID="adc3ea48-dff1-4bf1-8dfc-f134f768323b" containerID="9aba8f048cab0ad80ea46a64b75d387f7ce3d85012b40a7cca0d1b3d619a474e" exitCode=0 Jul 22 12:41:13.902654 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.902620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" event={"ID":"adc3ea48-dff1-4bf1-8dfc-f134f768323b","Type":"ContainerDied","Data":"9aba8f048cab0ad80ea46a64b75d387f7ce3d85012b40a7cca0d1b3d619a474e"} Jul 22 12:41:13.906178 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.906158 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:41:13.906473 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.906455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"1cb2b8f980ad4f1582dcea2534a9813c661753ac8a35e634928acd73795d59f2"} Jul 22 12:41:13.906644 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.906631 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:41:13.906739 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.906728 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:41:13.906844 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.906831 2576 scope.go:117] "RemoveContainer" containerID="f4b5ee189eb0c8694196803cf79be1675f26668a3949619b3e3ca144d7cd4fab" Jul 22 12:41:13.922069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:13.922052 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:41:14.911400 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:14.911375 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:41:14.912116 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:14.911724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" event={"ID":"e8f8c309-00be-466a-94db-f8ce110a007f","Type":"ContainerStarted","Data":"c6d3aab618996a3c383c22061aa409e9154624e648181af160b6187b27c9ec3c"} Jul 22 12:41:14.912116 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:14.912039 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:41:14.925824 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:14.925803 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:41:14.944297 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:14.944249 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" podStartSLOduration=8.79282706 podStartE2EDuration="28.944237531s" totalImagesPullingTime="20.151410471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:14.944121763 +0000 UTC m=+28.748838660" watchObservedRunningTime="2026-07-22 12:41:14.944237531 +0000 UTC m=+28.748954464" Jul 22 12:41:15.312215 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:15.312191 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4hvsk"] Jul 22 12:41:15.312361 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:15.312296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:15.312396 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:15.312380 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:15.315591 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:15.315565 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6wx8p"] Jul 22 12:41:15.315710 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:15.315665 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:15.315760 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:15.315744 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:15.316436 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:15.316411 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tn8h2"] Jul 22 12:41:15.316569 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:15.316526 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:15.316640 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:15.316625 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:15.915428 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:15.915398 2576 generic.go:356] "Generic (PLEG): container finished" podID="adc3ea48-dff1-4bf1-8dfc-f134f768323b" containerID="2237227e9dcd7ba2e8210bf15694665910d7702d49c3f8b83a9290f4e0a2a346" exitCode=0 Jul 22 12:41:15.915875 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:15.915486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" event={"ID":"adc3ea48-dff1-4bf1-8dfc-f134f768323b","Type":"ContainerDied","Data":"2237227e9dcd7ba2e8210bf15694665910d7702d49c3f8b83a9290f4e0a2a346"} Jul 22 12:41:16.708473 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:16.708444 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:16.708699 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:16.708541 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:17.707135 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:17.707106 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:17.707135 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:17.707141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:17.707498 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:17.707217 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:17.707498 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:17.707395 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:17.922552 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:17.922375 2576 generic.go:356] "Generic (PLEG): container finished" podID="adc3ea48-dff1-4bf1-8dfc-f134f768323b" containerID="2a8a8062e6c9b7120d341d8c5eb87a51f4125c474976bcf6f0cabdd302170ddf" exitCode=0 Jul 22 12:41:17.922552 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:17.922458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" event={"ID":"adc3ea48-dff1-4bf1-8dfc-f134f768323b","Type":"ContainerDied","Data":"2a8a8062e6c9b7120d341d8c5eb87a51f4125c474976bcf6f0cabdd302170ddf"} Jul 22 12:41:18.706722 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:18.706687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:18.706938 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:18.706834 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:19.326324 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:19.326288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:19.326713 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.326449 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:19.326713 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.326523 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs podName:d8f286d0-2bca-4211-9499-d80776bd4d35 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:51.326502429 +0000 UTC m=+65.131219307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs") pod "network-metrics-daemon-tn8h2" (UID: "d8f286d0-2bca-4211-9499-d80776bd4d35") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:19.427484 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:19.427451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:19.427615 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:19.427538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:19.427667 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.427620 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:19.427734 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.427669 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:19.427734 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.427685 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret podName:e10dae84-5ee9-4c3c-a86c-608de7b8504e nodeName:}" failed. No retries permitted until 2026-07-22 12:41:51.427671098 +0000 UTC m=+65.232387985 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret") pod "global-pull-secret-syncer-4hvsk" (UID: "e10dae84-5ee9-4c3c-a86c-608de7b8504e") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:19.427734 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.427686 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:19.427734 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.427699 2576 projected.go:196] Error preparing data for projected volume kube-api-access-zh46m for pod openshift-network-diagnostics/network-check-target-6wx8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:19.427734 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.427733 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m podName:ceea1b5d-f802-46e1-8391-111cfa88ed6d nodeName:}" failed. No retries permitted until 2026-07-22 12:41:51.427723409 +0000 UTC m=+65.232440311 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zh46m" (UniqueName: "kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m") pod "network-check-target-6wx8p" (UID: "ceea1b5d-f802-46e1-8391-111cfa88ed6d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:19.707427 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:19.707394 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:19.707427 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:19.707422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:19.707615 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.707537 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tn8h2" podUID="d8f286d0-2bca-4211-9499-d80776bd4d35" Jul 22 12:41:19.707668 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:19.707645 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6wx8p" podUID="ceea1b5d-f802-46e1-8391-111cfa88ed6d" Jul 22 12:41:20.706977 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:20.706942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:20.707469 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:20.707082 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4hvsk" podUID="e10dae84-5ee9-4c3c-a86c-608de7b8504e" Jul 22 12:41:21.049560 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.049493 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-88.ec2.internal" event="NodeReady" Jul 22 12:41:21.049714 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.049641 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 12:41:21.107314 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.107282 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57b85889f4-sqqhz"] Jul 22 12:41:21.109892 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.109871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.114887 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.114861 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 12:41:21.114887 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.114861 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fgszx\"" Jul 22 12:41:21.117574 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.117546 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 12:41:21.117694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.117585 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 12:41:21.122879 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.122828 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 12:41:21.131351 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.131331 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b85889f4-sqqhz"] Jul 22 12:41:21.131949 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.131930 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qkvwk"] Jul 22 12:41:21.134342 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.134322 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mpmlz"] Jul 22 12:41:21.134436 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.134410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.136216 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.136200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mpmlz" Jul 22 12:41:21.141589 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.141151 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 12:41:21.141589 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.141190 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 12:41:21.141589 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.141298 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kbrpx\"" Jul 22 12:41:21.141589 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.141422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 12:41:21.141589 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.141503 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 12:41:21.141589 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.141527 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 12:41:21.141937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.141747 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gqts7\"" Jul 22 12:41:21.164091 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.164063 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qkvwk"] Jul 22 12:41:21.164668 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.164645 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mpmlz"] Jul 22 12:41:21.241716 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.241851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-image-registry-private-configuration\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.241851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-ca-trust-extracted\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.241851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72cd096f-6bab-4c50-9088-186c56d2a701-config-volume\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.241851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bpc\" (UniqueName: \"kubernetes.io/projected/72cd096f-6bab-4c50-9088-186c56d2a701-kube-api-access-n2bpc\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.241851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhzj\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-kube-api-access-hxhzj\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.241851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.241851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrqm\" (UniqueName: \"kubernetes.io/projected/c729e819-258c-496d-befc-dca46019ce2c-kube-api-access-gqrqm\") pod \"ingress-canary-mpmlz\" (UID: \"c729e819-258c-496d-befc-dca46019ce2c\") " pod="openshift-ingress-canary/ingress-canary-mpmlz" Jul 22 12:41:21.242194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-certificates\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.242194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.241985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-installation-pull-secrets\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.242194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.242074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c729e819-258c-496d-befc-dca46019ce2c-cert\") pod \"ingress-canary-mpmlz\" (UID: \"c729e819-258c-496d-befc-dca46019ce2c\") " pod="openshift-ingress-canary/ingress-canary-mpmlz" Jul 22 12:41:21.242194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.242111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-trusted-ca\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.242194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.242141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-bound-sa-token\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.242194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.242164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cd096f-6bab-4c50-9088-186c56d2a701-tmp-dir\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.255643 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.255616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l9jtw_a1347662-9683-4412-9e63-61017d9d3be3/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:21.342937 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.342866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343087 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.342906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-image-registry-private-configuration\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343087 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.342974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-ca-trust-extracted\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343087 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.342997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72cd096f-6bab-4c50-9088-186c56d2a701-config-volume\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.343087 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.343055 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:21.343087 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.343070 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b85889f4-sqqhz: secret "image-registry-tls" not found Jul 22 12:41:21.343324 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.343139 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls podName:e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:21.843120604 +0000 UTC m=+35.647837478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls") pod "image-registry-57b85889f4-sqqhz" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4") : secret "image-registry-tls" not found Jul 22 12:41:21.343324 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bpc\" (UniqueName: \"kubernetes.io/projected/72cd096f-6bab-4c50-9088-186c56d2a701-kube-api-access-n2bpc\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.343324 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhzj\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-kube-api-access-hxhzj\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343324 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.343324 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrqm\" (UniqueName: \"kubernetes.io/projected/c729e819-258c-496d-befc-dca46019ce2c-kube-api-access-gqrqm\") pod \"ingress-canary-mpmlz\" (UID: \"c729e819-258c-496d-befc-dca46019ce2c\") " pod="openshift-ingress-canary/ingress-canary-mpmlz" Jul 22 12:41:21.343324 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-certificates\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343590 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-installation-pull-secrets\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343590 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c729e819-258c-496d-befc-dca46019ce2c-cert\") pod \"ingress-canary-mpmlz\" (UID: \"c729e819-258c-496d-befc-dca46019ce2c\") " pod="openshift-ingress-canary/ingress-canary-mpmlz" Jul 22 12:41:21.343590 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-ca-trust-extracted\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343590 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-trusted-ca\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343590 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.343419 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:21.343590 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-bound-sa-token\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.343590 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cd096f-6bab-4c50-9088-186c56d2a701-tmp-dir\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.343590 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.343466 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls podName:72cd096f-6bab-4c50-9088-186c56d2a701 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:21.843452024 +0000 UTC m=+35.648168901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls") pod "dns-default-qkvwk" (UID: "72cd096f-6bab-4c50-9088-186c56d2a701") : secret "dns-default-metrics-tls" not found Jul 22 12:41:21.343961 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343634 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72cd096f-6bab-4c50-9088-186c56d2a701-config-volume\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.343961 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.343662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72cd096f-6bab-4c50-9088-186c56d2a701-tmp-dir\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.344346 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.344324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-certificates\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.344502 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.344484 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-trusted-ca\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.347871 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.347850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-image-registry-private-configuration\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.347978 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.347872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-installation-pull-secrets\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.348052 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.347983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c729e819-258c-496d-befc-dca46019ce2c-cert\") pod \"ingress-canary-mpmlz\" (UID: \"c729e819-258c-496d-befc-dca46019ce2c\") " pod="openshift-ingress-canary/ingress-canary-mpmlz" Jul 22 12:41:21.357560 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.357534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l9jtw_a1347662-9683-4412-9e63-61017d9d3be3/konnectivity-agent/0.log" limit=1024 Jul 22 12:41:21.357665 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.357564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bpc\" (UniqueName: \"kubernetes.io/projected/72cd096f-6bab-4c50-9088-186c56d2a701-kube-api-access-n2bpc\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.357831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.357807 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-bound-sa-token\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.357974 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.357951 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrqm\" (UniqueName: \"kubernetes.io/projected/c729e819-258c-496d-befc-dca46019ce2c-kube-api-access-gqrqm\") pod \"ingress-canary-mpmlz\" (UID: \"c729e819-258c-496d-befc-dca46019ce2c\") " pod="openshift-ingress-canary/ingress-canary-mpmlz" Jul 22 12:41:21.358589 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.358548 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhzj\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-kube-api-access-hxhzj\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.451924 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.451894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mpmlz" Jul 22 12:41:21.707519 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.707483 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:21.707940 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.707669 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:21.710837 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.710785 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 12:41:21.710837 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.710835 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 12:41:21.711111 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.711093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-72nf2\"" Jul 22 12:41:21.711170 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.711093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv8qm\"" Jul 22 12:41:21.711170 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.711099 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 12:41:21.846991 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.846962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:21.847179 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:21.847004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:21.847179 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.847162 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:21.847179 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.847165 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:21.847179 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.847177 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b85889f4-sqqhz: secret "image-registry-tls" not found Jul 22 12:41:21.847367 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.847239 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls podName:e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:22.847219577 +0000 UTC m=+36.651936457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls") pod "image-registry-57b85889f4-sqqhz" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4") : secret "image-registry-tls" not found Jul 22 12:41:21.847367 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:21.847270 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls podName:72cd096f-6bab-4c50-9088-186c56d2a701 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:22.847245698 +0000 UTC m=+36.651962585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls") pod "dns-default-qkvwk" (UID: "72cd096f-6bab-4c50-9088-186c56d2a701") : secret "dns-default-metrics-tls" not found Jul 22 12:41:22.194485 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:22.194233 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mpmlz"] Jul 22 12:41:22.209813 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:22.209787 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc729e819_258c_496d_befc_dca46019ce2c.slice/crio-51eac00d800de0db892cfc91df9de44b5a91a81b021f3e80590a07785435a6a8 WatchSource:0}: Error finding container 51eac00d800de0db892cfc91df9de44b5a91a81b021f3e80590a07785435a6a8: Status 404 returned error can't find the container with id 51eac00d800de0db892cfc91df9de44b5a91a81b021f3e80590a07785435a6a8 Jul 22 12:41:22.706766 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:22.706737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:22.709576 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:22.709555 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 12:41:22.854927 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:22.854890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:22.855099 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:22.854944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:22.855099 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:22.855049 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:22.855099 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:22.855085 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b85889f4-sqqhz: secret "image-registry-tls" not found Jul 22 12:41:22.855099 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:22.855091 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:22.855293 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:22.855141 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls podName:72cd096f-6bab-4c50-9088-186c56d2a701 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:24.855128196 +0000 UTC m=+38.659845070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls") pod "dns-default-qkvwk" (UID: "72cd096f-6bab-4c50-9088-186c56d2a701") : secret "dns-default-metrics-tls" not found Jul 22 12:41:22.855293 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:22.855154 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls podName:e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:24.855148208 +0000 UTC m=+38.659865083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls") pod "image-registry-57b85889f4-sqqhz" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4") : secret "image-registry-tls" not found Jul 22 12:41:22.936205 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:22.936174 2576 generic.go:356] "Generic (PLEG): container finished" podID="adc3ea48-dff1-4bf1-8dfc-f134f768323b" containerID="475d68963c2726af91c561ff3d1a116ef1b8657e1d6dbeed6fa53e483ae4b51e" exitCode=0 Jul 22 12:41:22.936350 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:22.936254 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" event={"ID":"adc3ea48-dff1-4bf1-8dfc-f134f768323b","Type":"ContainerDied","Data":"475d68963c2726af91c561ff3d1a116ef1b8657e1d6dbeed6fa53e483ae4b51e"} Jul 22 12:41:22.937369 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:22.937347 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mpmlz" event={"ID":"c729e819-258c-496d-befc-dca46019ce2c","Type":"ContainerStarted","Data":"51eac00d800de0db892cfc91df9de44b5a91a81b021f3e80590a07785435a6a8"} Jul 22 12:41:23.943306 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:23.943265 2576 generic.go:356] "Generic (PLEG): container finished" podID="adc3ea48-dff1-4bf1-8dfc-f134f768323b" containerID="baad06e06005baf9a9cbdcfe4db5248e0f5456d7317fa70388b64926ef8fbf2e" exitCode=0 Jul 22 12:41:23.943906 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:23.943343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" event={"ID":"adc3ea48-dff1-4bf1-8dfc-f134f768323b","Type":"ContainerDied","Data":"baad06e06005baf9a9cbdcfe4db5248e0f5456d7317fa70388b64926ef8fbf2e"} Jul 22 12:41:24.871809 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:24.871648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:24.871899 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:24.871805 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:24.871899 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:24.871826 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b85889f4-sqqhz: secret "image-registry-tls" not found Jul 22 12:41:24.871899 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:24.871826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:24.871899 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:24.871884 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls podName:e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:28.871865122 +0000 UTC m=+42.676582002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls") pod "image-registry-57b85889f4-sqqhz" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4") : secret "image-registry-tls" not found Jul 22 12:41:24.872120 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:24.871927 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:24.872120 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:24.871969 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls podName:72cd096f-6bab-4c50-9088-186c56d2a701 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:28.871957736 +0000 UTC m=+42.676674610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls") pod "dns-default-qkvwk" (UID: "72cd096f-6bab-4c50-9088-186c56d2a701") : secret "dns-default-metrics-tls" not found Jul 22 12:41:24.947349 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:24.947321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mpmlz" event={"ID":"c729e819-258c-496d-befc-dca46019ce2c","Type":"ContainerStarted","Data":"20cf117cfa3cbdb6f71e1496b5c96e05837d75944c3c152e455afc728571f576"} Jul 22 12:41:24.950620 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:24.950596 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" event={"ID":"adc3ea48-dff1-4bf1-8dfc-f134f768323b","Type":"ContainerStarted","Data":"bf2192d81a17d1dcadf9137775a979f057e32f1d153cdfea5766e42ed79031a5"} Jul 22 12:41:24.968979 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:24.968938 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mpmlz" podStartSLOduration=1.558596385 podStartE2EDuration="3.968925868s" totalImagesPullingTime="2.410329483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:24.968274553 +0000 UTC m=+38.772991428" watchObservedRunningTime="2026-07-22 12:41:24.968925868 +0000 UTC m=+38.773642824" Jul 22 12:41:25.003659 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:25.003613 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zvfg7" podStartSLOduration=12.88096856 podStartE2EDuration="39.003595465s" totalImagesPullingTime="26.122626905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:25.002245915 +0000 UTC m=+38.806962811" watchObservedRunningTime="2026-07-22 12:41:25.003595465 +0000 UTC m=+38.808312361" Jul 22 12:41:26.015547 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.015517 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8d228"] Jul 22 12:41:26.035661 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.035637 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c964bf7f6-tqffj"] Jul 22 12:41:26.035818 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.035770 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.038503 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.038481 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 12:41:26.038630 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.038512 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 22 12:41:26.038630 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.038544 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 12:41:26.038630 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.038486 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lvlrb\"" Jul 22 12:41:26.038788 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.038728 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 22 12:41:26.044285 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.044269 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 22 12:41:26.059919 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.059893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8d228"] Jul 22 12:41:26.059919 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.059921 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c964bf7f6-tqffj"] Jul 22 12:41:26.060120 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.059956 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.062250 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.062228 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 22 12:41:26.062353 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.062320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 22 12:41:26.062403 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.062365 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 12:41:26.062649 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.062619 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 22 12:41:26.062649 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.062645 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 22 12:41:26.063047 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.063030 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-m85m6\"" Jul 22 12:41:26.063141 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.063125 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 12:41:26.120384 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.120354 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk"] Jul 22 12:41:26.140274 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.140251 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54"] Jul 22 12:41:26.140381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.140372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.144668 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.144646 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:41:26.144773 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.144672 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xdpnx\"" Jul 22 12:41:26.145074 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.145053 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 22 12:41:26.145180 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.145147 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 22 12:41:26.145250 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.145186 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 22 12:41:26.154042 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.154022 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w"] Jul 22 12:41:26.154126 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.154102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.157659 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.157642 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 22 12:41:26.158192 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.158172 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 22 12:41:26.158266 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.158237 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 22 12:41:26.159787 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.158674 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:41:26.159787 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.158888 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rkstm\"" Jul 22 12:41:26.172310 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.172291 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j"] Jul 22 12:41:26.172433 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.172418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w" Jul 22 12:41:26.175483 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.175469 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:41:26.175665 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.175652 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-lm7p7\"" Jul 22 12:41:26.175735 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.175715 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 22 12:41:26.181638 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/475631b0-ff58-405b-bdca-7228773d131d-snapshots\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.181712 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/475631b0-ff58-405b-bdca-7228773d131d-service-ca-bundle\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.181712 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.181712 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/475631b0-ff58-405b-bdca-7228773d131d-tmp\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.181826 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-stats-auth\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.181826 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181797 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5p57\" (UniqueName: \"kubernetes.io/projected/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-kube-api-access-h5p57\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.181826 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/475631b0-ff58-405b-bdca-7228773d131d-serving-cert\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.181928 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-default-certificate\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.181928 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.181928 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/475631b0-ff58-405b-bdca-7228773d131d-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.182072 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.181930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msspg\" (UniqueName: \"kubernetes.io/projected/475631b0-ff58-405b-bdca-7228773d131d-kube-api-access-msspg\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.198362 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.198343 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-rck2w"] Jul 22 12:41:26.198448 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.198439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.202550 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.202533 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 22 12:41:26.203136 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.203120 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 12:41:26.203228 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.203208 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 22 12:41:26.203480 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.203467 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 12:41:26.204175 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.204159 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q4jdp\"" Jul 22 12:41:26.221852 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.221832 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk"] Jul 22 12:41:26.221923 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.221855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54"] Jul 22 12:41:26.221923 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.221869 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j"] Jul 22 12:41:26.221923 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.221880 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w"] Jul 22 12:41:26.221923 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.221898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-rck2w"] Jul 22 12:41:26.222094 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.221931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.224118 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.224094 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:41:26.225160 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.225143 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 22 12:41:26.225238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.225182 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m269c\"" Jul 22 12:41:26.225466 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.225450 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 22 12:41:26.227643 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.227625 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 22 12:41:26.232750 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.232734 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 22 12:41:26.241787 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.241768 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r"] Jul 22 12:41:26.263876 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.263859 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r"] Jul 22 12:41:26.263939 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.263929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:26.266226 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.266178 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 22 12:41:26.266367 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.266344 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 22 12:41:26.266441 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.266380 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:41:26.266441 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.266391 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cvw7c\"" Jul 22 12:41:26.282436 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.282416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5p57\" (UniqueName: \"kubernetes.io/projected/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-kube-api-access-h5p57\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.282515 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.282441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-default-certificate\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.282515 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.282457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.282515 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.282477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/475631b0-ff58-405b-bdca-7228773d131d-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.282607 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.282521 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msspg\" (UniqueName: \"kubernetes.io/projected/475631b0-ff58-405b-bdca-7228773d131d-kube-api-access-msspg\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.282607 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.282566 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 12:41:26.282967 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.282904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrb7\" (UniqueName: \"kubernetes.io/projected/749ddc12-d2c3-4c26-aede-b69956acd623-kube-api-access-zzrb7\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.283193 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.282994 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:26.782948435 +0000 UTC m=+40.587665311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : secret "router-metrics-certs-default" not found Jul 22 12:41:26.283691 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.283210 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8x49\" (UniqueName: \"kubernetes.io/projected/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-kube-api-access-s8x49\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.283837 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.283732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/749ddc12-d2c3-4c26-aede-b69956acd623-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.284068 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.284049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/475631b0-ff58-405b-bdca-7228773d131d-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.284135 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.284065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/749ddc12-d2c3-4c26-aede-b69956acd623-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.284228 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.284212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b465\" (UniqueName: \"kubernetes.io/projected/e61b8b11-07b7-413a-87e5-525e3a5f230d-kube-api-access-9b465\") pod \"volume-data-source-validator-78cf4c6f6-jwg8w\" (UID: \"e61b8b11-07b7-413a-87e5-525e3a5f230d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w" Jul 22 12:41:26.284417 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.284364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-stats-auth\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.284585 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.284539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/475631b0-ff58-405b-bdca-7228773d131d-serving-cert\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.284788 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.284685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/475631b0-ff58-405b-bdca-7228773d131d-service-ca-bundle\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.284894 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.284817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/475631b0-ff58-405b-bdca-7228773d131d-snapshots\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.284950 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.284923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.285052 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.285001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.285104 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.285085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/475631b0-ff58-405b-bdca-7228773d131d-tmp\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.285172 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.285158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-config\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.286856 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.286835 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:26.786816621 +0000 UTC m=+40.591533510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : configmap references non-existent config key: service-ca.crt Jul 22 12:41:26.287369 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.287347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/475631b0-ff58-405b-bdca-7228773d131d-service-ca-bundle\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.287869 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.287848 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/475631b0-ff58-405b-bdca-7228773d131d-snapshots\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.288889 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.288871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/475631b0-ff58-405b-bdca-7228773d131d-serving-cert\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.289130 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.289110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-stats-auth\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.289181 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.289110 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-default-certificate\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.292399 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.292375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msspg\" (UniqueName: \"kubernetes.io/projected/475631b0-ff58-405b-bdca-7228773d131d-kube-api-access-msspg\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.292460 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.292421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5p57\" (UniqueName: \"kubernetes.io/projected/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-kube-api-access-h5p57\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.299702 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.299685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/475631b0-ff58-405b-bdca-7228773d131d-tmp\") pod \"insights-operator-555d9d7885-8d228\" (UID: \"475631b0-ff58-405b-bdca-7228773d131d\") " pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.345933 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.345908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-8d228" Jul 22 12:41:26.386218 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.386321 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m62cv\" (UniqueName: \"kubernetes.io/projected/f033188f-9515-4e0f-96c3-d0bf016e6a46-kube-api-access-m62cv\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:26.386321 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386246 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csckq\" (UniqueName: \"kubernetes.io/projected/0550d49e-243e-437e-b4a6-5badb236b4fe-kube-api-access-csckq\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.386321 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrb7\" (UniqueName: \"kubernetes.io/projected/749ddc12-d2c3-4c26-aede-b69956acd623-kube-api-access-zzrb7\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.386464 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8x49\" (UniqueName: \"kubernetes.io/projected/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-kube-api-access-s8x49\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.386517 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/749ddc12-d2c3-4c26-aede-b69956acd623-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.386564 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386514 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0550d49e-243e-437e-b4a6-5badb236b4fe-trusted-ca\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.386648 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/749ddc12-d2c3-4c26-aede-b69956acd623-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.386704 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0550d49e-243e-437e-b4a6-5badb236b4fe-config\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.386759 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b465\" (UniqueName: \"kubernetes.io/projected/e61b8b11-07b7-413a-87e5-525e3a5f230d-kube-api-access-9b465\") pod \"volume-data-source-validator-78cf4c6f6-jwg8w\" (UID: \"e61b8b11-07b7-413a-87e5-525e3a5f230d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w" Jul 22 12:41:26.386848 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httvl\" (UniqueName: \"kubernetes.io/projected/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-kube-api-access-httvl\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.386917 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386847 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0550d49e-243e-437e-b4a6-5badb236b4fe-serving-cert\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.386969 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.386969 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:26.387097 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.387097 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.386997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/749ddc12-d2c3-4c26-aede-b69956acd623-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.387097 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.387046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-config\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.387634 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.387612 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-config\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.388955 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.388927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/749ddc12-d2c3-4c26-aede-b69956acd623-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.389379 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.389363 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-serving-cert\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.396218 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.396201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8x49\" (UniqueName: \"kubernetes.io/projected/0701213e-6c4b-41aa-aac6-38e0f6cc47f9-kube-api-access-s8x49\") pod \"service-ca-operator-5bcbc94c78-hgzzk\" (UID: \"0701213e-6c4b-41aa-aac6-38e0f6cc47f9\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.396323 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.396296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrb7\" (UniqueName: \"kubernetes.io/projected/749ddc12-d2c3-4c26-aede-b69956acd623-kube-api-access-zzrb7\") pod \"kube-storage-version-migrator-operator-7658d9b749-n2m54\" (UID: \"749ddc12-d2c3-4c26-aede-b69956acd623\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.397365 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.397332 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b465\" (UniqueName: \"kubernetes.io/projected/e61b8b11-07b7-413a-87e5-525e3a5f230d-kube-api-access-9b465\") pod \"volume-data-source-validator-78cf4c6f6-jwg8w\" (UID: \"e61b8b11-07b7-413a-87e5-525e3a5f230d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w" Jul 22 12:41:26.449303 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.449260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" Jul 22 12:41:26.466118 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.466090 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" Jul 22 12:41:26.480108 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.480087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w" Jul 22 12:41:26.485132 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.485112 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5"] Jul 22 12:41:26.488113 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0550d49e-243e-437e-b4a6-5badb236b4fe-trusted-ca\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.488202 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488130 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0550d49e-243e-437e-b4a6-5badb236b4fe-config\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.488307 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httvl\" (UniqueName: \"kubernetes.io/projected/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-kube-api-access-httvl\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.488357 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0550d49e-243e-437e-b4a6-5badb236b4fe-serving-cert\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.488463 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.488513 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:26.488878 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.488878 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62cv\" (UniqueName: \"kubernetes.io/projected/f033188f-9515-4e0f-96c3-d0bf016e6a46-kube-api-access-m62cv\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:26.488878 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.488629 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:26.488878 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csckq\" (UniqueName: \"kubernetes.io/projected/0550d49e-243e-437e-b4a6-5badb236b4fe-kube-api-access-csckq\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.488878 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.488663 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 12:41:26.488878 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.488704 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls podName:f68990c1-be9f-4a0a-9165-cb17dcfa2ec3 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:26.98868634 +0000 UTC m=+40.793403227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8fq6j" (UID: "f68990c1-be9f-4a0a-9165-cb17dcfa2ec3") : secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:26.488878 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.488757 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls podName:f033188f-9515-4e0f-96c3-d0bf016e6a46 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:26.988744233 +0000 UTC m=+40.793461108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-vdn7r" (UID: "f033188f-9515-4e0f-96c3-d0bf016e6a46") : secret "samples-operator-tls" not found Jul 22 12:41:26.488878 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0550d49e-243e-437e-b4a6-5badb236b4fe-config\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.489246 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.488995 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0550d49e-243e-437e-b4a6-5badb236b4fe-trusted-ca\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.489290 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.489246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.490693 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.490677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0550d49e-243e-437e-b4a6-5badb236b4fe-serving-cert\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.507884 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.507861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-httvl\" (UniqueName: \"kubernetes.io/projected/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-kube-api-access-httvl\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.508861 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:26.508830 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475631b0_ff58_405b_bdca_7228773d131d.slice/crio-d2fc570872411d2f5776be4c590ac29913a5cb1b3674d58486912c66959889a0 WatchSource:0}: Error finding container d2fc570872411d2f5776be4c590ac29913a5cb1b3674d58486912c66959889a0: Status 404 returned error can't find the container with id d2fc570872411d2f5776be4c590ac29913a5cb1b3674d58486912c66959889a0 Jul 22 12:41:26.508957 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.508922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csckq\" (UniqueName: \"kubernetes.io/projected/0550d49e-243e-437e-b4a6-5badb236b4fe-kube-api-access-csckq\") pod \"console-operator-565b7b569d-rck2w\" (UID: \"0550d49e-243e-437e-b4a6-5badb236b4fe\") " pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.508957 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.508934 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62cv\" (UniqueName: \"kubernetes.io/projected/f033188f-9515-4e0f-96c3-d0bf016e6a46-kube-api-access-m62cv\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:26.514258 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.514213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5"] Jul 22 12:41:26.514258 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.514242 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-8d228"] Jul 22 12:41:26.514418 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.514316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5" Jul 22 12:41:26.519090 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.517070 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jjkvf\"" Jul 22 12:41:26.529496 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.529474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:26.611870 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.611802 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk"] Jul 22 12:41:26.615551 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:26.615524 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0701213e_6c4b_41aa_aac6_38e0f6cc47f9.slice/crio-df2caae3b3e0474a4b1cc4d848d3d29991955da29bd56c0b53944bfe394374a3 WatchSource:0}: Error finding container df2caae3b3e0474a4b1cc4d848d3d29991955da29bd56c0b53944bfe394374a3: Status 404 returned error can't find the container with id df2caae3b3e0474a4b1cc4d848d3d29991955da29bd56c0b53944bfe394374a3 Jul 22 12:41:26.634148 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:26.634114 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749ddc12_d2c3_4c26_aede_b69956acd623.slice/crio-a4d60339c94bb17c753d0f3ac8b35f19ce1dd93972ef382f1f54eb16ea8cc822 WatchSource:0}: Error finding container a4d60339c94bb17c753d0f3ac8b35f19ce1dd93972ef382f1f54eb16ea8cc822: Status 404 returned error can't find the container with id a4d60339c94bb17c753d0f3ac8b35f19ce1dd93972ef382f1f54eb16ea8cc822 Jul 22 12:41:26.634570 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.634515 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54"] Jul 22 12:41:26.678124 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.678100 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w"] Jul 22 12:41:26.679770 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:26.679746 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61b8b11_07b7_413a_87e5_525e3a5f230d.slice/crio-4fd93568bd9eb2e07322db493f9523d11cdbb2d94437bbc7e8f95c3693e9d10b WatchSource:0}: Error finding container 4fd93568bd9eb2e07322db493f9523d11cdbb2d94437bbc7e8f95c3693e9d10b: Status 404 returned error can't find the container with id 4fd93568bd9eb2e07322db493f9523d11cdbb2d94437bbc7e8f95c3693e9d10b Jul 22 12:41:26.691939 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.691911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f822w\" (UniqueName: \"kubernetes.io/projected/67eb92cb-6da6-479e-9d49-6eda88451c77-kube-api-access-f822w\") pod \"network-check-source-7bc5b6bf85-b4tq5\" (UID: \"67eb92cb-6da6-479e-9d49-6eda88451c77\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5" Jul 22 12:41:26.711249 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:26.711228 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0550d49e_243e_437e_b4a6_5badb236b4fe.slice/crio-dca2259abba0f6223cde965d959923f5e15c0f53c196b8dd2e68ad8165ae4e70 WatchSource:0}: Error finding container dca2259abba0f6223cde965d959923f5e15c0f53c196b8dd2e68ad8165ae4e70: Status 404 returned error can't find the container with id dca2259abba0f6223cde965d959923f5e15c0f53c196b8dd2e68ad8165ae4e70 Jul 22 12:41:26.711388 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.711369 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-rck2w"] Jul 22 12:41:26.792707 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.792661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.792707 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.792692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:26.792828 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.792739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f822w\" (UniqueName: \"kubernetes.io/projected/67eb92cb-6da6-479e-9d49-6eda88451c77-kube-api-access-f822w\") pod \"network-check-source-7bc5b6bf85-b4tq5\" (UID: \"67eb92cb-6da6-479e-9d49-6eda88451c77\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5" Jul 22 12:41:26.792828 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.792818 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:27.792798352 +0000 UTC m=+41.597515227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : configmap references non-existent config key: service-ca.crt Jul 22 12:41:26.792897 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.792875 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 12:41:26.792936 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.792926 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:27.792910538 +0000 UTC m=+41.597627428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : secret "router-metrics-certs-default" not found Jul 22 12:41:26.801454 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.801436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f822w\" (UniqueName: \"kubernetes.io/projected/67eb92cb-6da6-479e-9d49-6eda88451c77-kube-api-access-f822w\") pod \"network-check-source-7bc5b6bf85-b4tq5\" (UID: \"67eb92cb-6da6-479e-9d49-6eda88451c77\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5" Jul 22 12:41:26.827241 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.827226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5" Jul 22 12:41:26.957256 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.957223 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" event={"ID":"0550d49e-243e-437e-b4a6-5badb236b4fe","Type":"ContainerStarted","Data":"dca2259abba0f6223cde965d959923f5e15c0f53c196b8dd2e68ad8165ae4e70"} Jul 22 12:41:26.958239 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.958215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w" event={"ID":"e61b8b11-07b7-413a-87e5-525e3a5f230d","Type":"ContainerStarted","Data":"4fd93568bd9eb2e07322db493f9523d11cdbb2d94437bbc7e8f95c3693e9d10b"} Jul 22 12:41:26.959201 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.959177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" event={"ID":"749ddc12-d2c3-4c26-aede-b69956acd623","Type":"ContainerStarted","Data":"a4d60339c94bb17c753d0f3ac8b35f19ce1dd93972ef382f1f54eb16ea8cc822"} Jul 22 12:41:26.960199 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.960179 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" event={"ID":"0701213e-6c4b-41aa-aac6-38e0f6cc47f9","Type":"ContainerStarted","Data":"df2caae3b3e0474a4b1cc4d848d3d29991955da29bd56c0b53944bfe394374a3"} Jul 22 12:41:26.961200 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.961171 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8d228" event={"ID":"475631b0-ff58-405b-bdca-7228773d131d","Type":"ContainerStarted","Data":"d2fc570872411d2f5776be4c590ac29913a5cb1b3674d58486912c66959889a0"} Jul 22 12:41:26.994714 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.994688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:26.994796 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:26.994723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:26.994860 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.994827 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:26.994860 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.994839 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 12:41:26.994947 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.994878 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls podName:f68990c1-be9f-4a0a-9165-cb17dcfa2ec3 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:27.99486409 +0000 UTC m=+41.799580965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8fq6j" (UID: "f68990c1-be9f-4a0a-9165-cb17dcfa2ec3") : secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:26.994947 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:26.994895 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls podName:f033188f-9515-4e0f-96c3-d0bf016e6a46 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:27.994885227 +0000 UTC m=+41.799602103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-vdn7r" (UID: "f033188f-9515-4e0f-96c3-d0bf016e6a46") : secret "samples-operator-tls" not found Jul 22 12:41:27.000996 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:27.000978 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5"] Jul 22 12:41:27.003750 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:27.003726 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67eb92cb_6da6_479e_9d49_6eda88451c77.slice/crio-8d5f9f9accc8b2e02d32512fdd62113c8b045c49ee4b030262e8a6ce4dde60ec WatchSource:0}: Error finding container 8d5f9f9accc8b2e02d32512fdd62113c8b045c49ee4b030262e8a6ce4dde60ec: Status 404 returned error can't find the container with id 8d5f9f9accc8b2e02d32512fdd62113c8b045c49ee4b030262e8a6ce4dde60ec Jul 22 12:41:27.804869 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:27.804835 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:27.805330 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:27.805031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:27.805330 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:27.805177 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:29.805157639 +0000 UTC m=+43.609874536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : configmap references non-existent config key: service-ca.crt Jul 22 12:41:27.805572 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:27.805554 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 12:41:27.805631 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:27.805615 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:29.805600341 +0000 UTC m=+43.610317221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : secret "router-metrics-certs-default" not found Jul 22 12:41:27.966252 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:27.966105 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5" event={"ID":"67eb92cb-6da6-479e-9d49-6eda88451c77","Type":"ContainerStarted","Data":"8d5f9f9accc8b2e02d32512fdd62113c8b045c49ee4b030262e8a6ce4dde60ec"} Jul 22 12:41:28.008187 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:28.007297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:28.008187 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:28.007346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:28.008187 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.007617 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 12:41:28.008187 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.007677 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls podName:f033188f-9515-4e0f-96c3-d0bf016e6a46 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:30.007660034 +0000 UTC m=+43.812376908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-vdn7r" (UID: "f033188f-9515-4e0f-96c3-d0bf016e6a46") : secret "samples-operator-tls" not found Jul 22 12:41:28.008187 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.008101 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:28.008187 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.008154 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls podName:f68990c1-be9f-4a0a-9165-cb17dcfa2ec3 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:30.008138144 +0000 UTC m=+43.812855022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8fq6j" (UID: "f68990c1-be9f-4a0a-9165-cb17dcfa2ec3") : secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:28.917032 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:28.916976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:28.917438 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.917157 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:28.917438 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:28.917196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:28.917438 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.917238 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls podName:72cd096f-6bab-4c50-9088-186c56d2a701 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:36.91721649 +0000 UTC m=+50.721933382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls") pod "dns-default-qkvwk" (UID: "72cd096f-6bab-4c50-9088-186c56d2a701") : secret "dns-default-metrics-tls" not found Jul 22 12:41:28.917438 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.917306 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:28.917438 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.917322 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b85889f4-sqqhz: secret "image-registry-tls" not found Jul 22 12:41:28.917438 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:28.917377 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls podName:e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:36.917362362 +0000 UTC m=+50.722079237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls") pod "image-registry-57b85889f4-sqqhz" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4") : secret "image-registry-tls" not found Jul 22 12:41:29.824795 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:29.824765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:29.824954 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:29.824884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:29.824954 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:29.824915 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 12:41:29.825055 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:29.824978 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:33.824965311 +0000 UTC m=+47.629682185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : configmap references non-existent config key: service-ca.crt Jul 22 12:41:29.825055 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:29.824992 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:33.824986933 +0000 UTC m=+47.629703807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : secret "router-metrics-certs-default" not found Jul 22 12:41:30.026486 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:30.026453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:30.026486 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:30.026488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:30.026909 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:30.026630 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:30.026909 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:30.026686 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 12:41:30.026909 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:30.026711 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls podName:f68990c1-be9f-4a0a-9165-cb17dcfa2ec3 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:34.026688999 +0000 UTC m=+47.831405875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8fq6j" (UID: "f68990c1-be9f-4a0a-9165-cb17dcfa2ec3") : secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:30.026909 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:30.026733 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls podName:f033188f-9515-4e0f-96c3-d0bf016e6a46 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:34.02671843 +0000 UTC m=+47.831435308 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-vdn7r" (UID: "f033188f-9515-4e0f-96c3-d0bf016e6a46") : secret "samples-operator-tls" not found Jul 22 12:41:32.989213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.989172 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8d228" event={"ID":"475631b0-ff58-405b-bdca-7228773d131d","Type":"ContainerStarted","Data":"9c22b6efa943b1f46ea1c6ac113aec80feb90e2e77ca5ad6469ce8370b2fcf26"} Jul 22 12:41:32.990923 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.990896 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/0.log" Jul 22 12:41:32.991045 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.990938 2576 generic.go:356] "Generic (PLEG): container finished" podID="0550d49e-243e-437e-b4a6-5badb236b4fe" containerID="77117c36c52aa254598af0a8dbee299ee6cd24ca2b6f579f7a1ff311edcce6ef" exitCode=255 Jul 22 12:41:32.991045 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.990968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" event={"ID":"0550d49e-243e-437e-b4a6-5badb236b4fe","Type":"ContainerDied","Data":"77117c36c52aa254598af0a8dbee299ee6cd24ca2b6f579f7a1ff311edcce6ef"} Jul 22 12:41:32.991474 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.991458 2576 scope.go:117] "RemoveContainer" containerID="77117c36c52aa254598af0a8dbee299ee6cd24ca2b6f579f7a1ff311edcce6ef" Jul 22 12:41:32.993646 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.993479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w" event={"ID":"e61b8b11-07b7-413a-87e5-525e3a5f230d","Type":"ContainerStarted","Data":"1d4e8e5b85d4920d34a0e7f5065ea0c1a13061a52c0bad8109db553915d9927d"} Jul 22 12:41:32.995077 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.995056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" event={"ID":"749ddc12-d2c3-4c26-aede-b69956acd623","Type":"ContainerStarted","Data":"0b999f80c1becb041a2798cf63061af97529077a8e5fd81e5d5c5b9abd375c62"} Jul 22 12:41:32.996889 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.996867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5" event={"ID":"67eb92cb-6da6-479e-9d49-6eda88451c77","Type":"ContainerStarted","Data":"bb3df2ae5207c341ca1e47dcb5dcefacfaab16161b330fb1feb848a0437a25c6"} Jul 22 12:41:32.998451 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:32.998431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" event={"ID":"0701213e-6c4b-41aa-aac6-38e0f6cc47f9","Type":"ContainerStarted","Data":"38896578e0e92e15de4a4d58ced7a52acbbf37cf7d7970580b251fbfec94bc51"} Jul 22 12:41:33.016994 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.016387 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-8d228" podStartSLOduration=0.892395302 podStartE2EDuration="7.016376588s" totalImagesPullingTime="6.123981286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:33.015201402 +0000 UTC m=+46.819918299" watchObservedRunningTime="2026-07-22 12:41:33.016376588 +0000 UTC m=+46.821093484" Jul 22 12:41:33.054054 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.051969 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" podStartSLOduration=1.047810268 podStartE2EDuration="7.051952734s" totalImagesPullingTime="6.004142466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:33.050483887 +0000 UTC m=+46.855200783" watchObservedRunningTime="2026-07-22 12:41:33.051952734 +0000 UTC m=+46.856669631" Jul 22 12:41:33.081919 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.081869 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jwg8w" podStartSLOduration=1.199417606 podStartE2EDuration="7.081852283s" totalImagesPullingTime="5.882434677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:33.081161622 +0000 UTC m=+46.885878519" watchObservedRunningTime="2026-07-22 12:41:33.081852283 +0000 UTC m=+46.886569158" Jul 22 12:41:33.128081 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.128030 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-b4tq5" podStartSLOduration=1.498035441 podStartE2EDuration="7.127996285s" totalImagesPullingTime="5.629960844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:33.125687985 +0000 UTC m=+46.930404881" watchObservedRunningTime="2026-07-22 12:41:33.127996285 +0000 UTC m=+46.932713182" Jul 22 12:41:33.678044 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.676516 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" podStartSLOduration=1.658785202 podStartE2EDuration="7.676491022s" totalImagesPullingTime="6.01770582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:33.150338517 +0000 UTC m=+46.955055413" watchObservedRunningTime="2026-07-22 12:41:33.676491022 +0000 UTC m=+47.481208109" Jul 22 12:41:33.678044 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.677147 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rb9sd"] Jul 22 12:41:33.696847 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.696822 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.699891 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.699706 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mvgjr\"" Jul 22 12:41:33.700182 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.700166 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 12:41:33.700838 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.700813 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rb9sd"] Jul 22 12:41:33.701118 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.701094 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 12:41:33.758161 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.758137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.758262 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.758230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr68f\" (UniqueName: \"kubernetes.io/projected/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-api-access-wr68f\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.758406 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.758385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.758483 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.758460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-crio-socket\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.758533 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.758519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-data-volume\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.859652 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.859616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr68f\" (UniqueName: \"kubernetes.io/projected/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-api-access-wr68f\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.859791 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.859746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.859848 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.859798 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:33.859848 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.859826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-crio-socket\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.859950 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.859854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:33.859950 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.859881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-data-volume\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.859950 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.859908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.860118 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:33.859997 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:33.860118 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:33.860084 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:34.360063382 +0000 UTC m=+48.164780271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : configmap "kube-rbac-proxy" not found Jul 22 12:41:33.860227 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.860145 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-crio-socket\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.860278 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:33.860266 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:33.860342 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:33.860315 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 12:41:33.860392 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:33.860318 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:34.360301292 +0000 UTC m=+48.165018175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:33.860392 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.860379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-data-volume\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:33.860494 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:33.860403 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:41.860379746 +0000 UTC m=+55.665096621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : configmap references non-existent config key: service-ca.crt Jul 22 12:41:33.860494 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:33.860426 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:41.860412752 +0000 UTC m=+55.665129630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : secret "router-metrics-certs-default" not found Jul 22 12:41:33.874702 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:33.874683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr68f\" (UniqueName: \"kubernetes.io/projected/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-api-access-wr68f\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:34.004263 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.004191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:41:34.004678 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.004660 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/0.log" Jul 22 12:41:34.004754 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.004705 2576 generic.go:356] "Generic (PLEG): container finished" podID="0550d49e-243e-437e-b4a6-5badb236b4fe" containerID="df7eaea6d64ec8d8e0ecb41a23989f6b599bc987bda028fcd10c5fd8d06419b6" exitCode=255 Jul 22 12:41:34.004830 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.004806 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" event={"ID":"0550d49e-243e-437e-b4a6-5badb236b4fe","Type":"ContainerDied","Data":"df7eaea6d64ec8d8e0ecb41a23989f6b599bc987bda028fcd10c5fd8d06419b6"} Jul 22 12:41:34.004885 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.004863 2576 scope.go:117] "RemoveContainer" containerID="77117c36c52aa254598af0a8dbee299ee6cd24ca2b6f579f7a1ff311edcce6ef" Jul 22 12:41:34.005222 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.005051 2576 scope.go:117] "RemoveContainer" containerID="df7eaea6d64ec8d8e0ecb41a23989f6b599bc987bda028fcd10c5fd8d06419b6" Jul 22 12:41:34.005330 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.005245 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-rck2w_openshift-console-operator(0550d49e-243e-437e-b4a6-5badb236b4fe)\"" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" podUID="0550d49e-243e-437e-b4a6-5badb236b4fe" Jul 22 12:41:34.061308 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.061276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:34.061308 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.061308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:34.061483 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.061411 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:34.061483 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.061433 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 22 12:41:34.061483 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.061469 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls podName:f68990c1-be9f-4a0a-9165-cb17dcfa2ec3 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:42.061451598 +0000 UTC m=+55.866168496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8fq6j" (UID: "f68990c1-be9f-4a0a-9165-cb17dcfa2ec3") : secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:34.061637 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.061513 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls podName:f033188f-9515-4e0f-96c3-d0bf016e6a46 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:42.061495693 +0000 UTC m=+55.866212569 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-vdn7r" (UID: "f033188f-9515-4e0f-96c3-d0bf016e6a46") : secret "samples-operator-tls" not found Jul 22 12:41:34.365154 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.365078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:34.365299 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:34.365163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:34.365299 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.365231 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:34.365299 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.365258 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:34.365299 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.365299 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:35.365284444 +0000 UTC m=+49.170001319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:34.365437 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:34.365316 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:35.365308525 +0000 UTC m=+49.170025400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : configmap "kube-rbac-proxy" not found Jul 22 12:41:35.009813 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:35.009778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:41:35.010230 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:35.010168 2576 scope.go:117] "RemoveContainer" containerID="df7eaea6d64ec8d8e0ecb41a23989f6b599bc987bda028fcd10c5fd8d06419b6" Jul 22 12:41:35.010353 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:35.010327 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-rck2w_openshift-console-operator(0550d49e-243e-437e-b4a6-5badb236b4fe)\"" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" podUID="0550d49e-243e-437e-b4a6-5badb236b4fe" Jul 22 12:41:35.266180 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:35.266109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qh6q7_713f7499-9b3a-4f9c-b6a8-8682fb8f507d/dns-node-resolver/0.log" Jul 22 12:41:35.372899 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:35.372871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:35.373059 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:35.372959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:35.373059 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:35.373007 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:35.373138 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:35.373075 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:37.373060464 +0000 UTC m=+51.177777338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:35.373138 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:35.373076 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:35.373138 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:35.373120 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:37.373108905 +0000 UTC m=+51.177825782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : configmap "kube-rbac-proxy" not found Jul 22 12:41:36.064601 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:36.064573 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7kfr8_cdab67a7-9489-4d5c-ad85-098e8a5c7c7d/node-ca/0.log" Jul 22 12:41:36.530649 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:36.530613 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:36.530821 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:36.530712 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:36.531092 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:36.531075 2576 scope.go:117] "RemoveContainer" containerID="df7eaea6d64ec8d8e0ecb41a23989f6b599bc987bda028fcd10c5fd8d06419b6" Jul 22 12:41:36.531259 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:36.531239 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-rck2w_openshift-console-operator(0550d49e-243e-437e-b4a6-5badb236b4fe)\"" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" podUID="0550d49e-243e-437e-b4a6-5badb236b4fe" Jul 22 12:41:36.987728 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:36.987698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:36.987728 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:36.987731 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:36.987919 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:36.987821 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:36.987919 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:36.987841 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:36.987919 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:36.987859 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b85889f4-sqqhz: secret "image-registry-tls" not found Jul 22 12:41:36.987919 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:36.987871 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls podName:72cd096f-6bab-4c50-9088-186c56d2a701 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:52.987859092 +0000 UTC m=+66.792575966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls") pod "dns-default-qkvwk" (UID: "72cd096f-6bab-4c50-9088-186c56d2a701") : secret "dns-default-metrics-tls" not found Jul 22 12:41:36.987919 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:36.987904 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls podName:e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:52.987889757 +0000 UTC m=+66.792606635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls") pod "image-registry-57b85889f4-sqqhz" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4") : secret "image-registry-tls" not found Jul 22 12:41:37.066437 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:37.066390 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mpmlz_c729e819-258c-496d-befc-dca46019ce2c/serve-healthcheck-canary/0.log" Jul 22 12:41:37.391460 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:37.391392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:37.391590 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:37.391459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:37.391590 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:37.391524 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:37.391590 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:37.391536 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:37.391590 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:37.391577 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:41.391564448 +0000 UTC m=+55.196281326 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : configmap "kube-rbac-proxy" not found Jul 22 12:41:37.391590 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:37.391589 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:41.391583823 +0000 UTC m=+55.196300697 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:41.423048 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:41.422996 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:41.423481 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:41.423115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:41.423481 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:41.423156 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:41.423481 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:41.423219 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls podName:ef7f534d-ea89-4086-8ad1-d7a303cc8ec8 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:49.423205019 +0000 UTC m=+63.227921893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rb9sd" (UID: "ef7f534d-ea89-4086-8ad1-d7a303cc8ec8") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:41.423606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:41.423584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:41.927768 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:41.927736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:41.928061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:41.927783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:41.928061 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:41.927899 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:57.927880726 +0000 UTC m=+71.732597600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : configmap references non-existent config key: service-ca.crt Jul 22 12:41:41.928061 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:41.927938 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 22 12:41:41.928061 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:41.928031 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs podName:5e7a2923-60f8-434a-9cc3-2cc73825ddcf nodeName:}" failed. No retries permitted until 2026-07-22 12:41:57.927991678 +0000 UTC m=+71.732708558 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs") pod "router-default-c964bf7f6-tqffj" (UID: "5e7a2923-60f8-434a-9cc3-2cc73825ddcf") : secret "router-metrics-certs-default" not found Jul 22 12:41:42.129296 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:42.129263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:42.129296 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:42.129294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:42.129501 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:42.129402 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:42.129501 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:41:42.129466 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls podName:f68990c1-be9f-4a0a-9165-cb17dcfa2ec3 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:58.129452511 +0000 UTC m=+71.934169385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-8fq6j" (UID: "f68990c1-be9f-4a0a-9165-cb17dcfa2ec3") : secret "cluster-monitoring-operator-tls" not found Jul 22 12:41:42.131760 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:42.131730 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f033188f-9515-4e0f-96c3-d0bf016e6a46-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-vdn7r\" (UID: \"f033188f-9515-4e0f-96c3-d0bf016e6a46\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:42.171828 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:42.171805 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" Jul 22 12:41:42.292102 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:42.292074 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r"] Jul 22 12:41:43.035544 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:43.035513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" event={"ID":"f033188f-9515-4e0f-96c3-d0bf016e6a46","Type":"ContainerStarted","Data":"25c20d5e4525e3713214997631de68f26a179276f6623b9ed14631a7b100bbbc"} Jul 22 12:41:44.039943 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:44.039909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" event={"ID":"f033188f-9515-4e0f-96c3-d0bf016e6a46","Type":"ContainerStarted","Data":"0a2c380f762f69d5d2686577898133cf84fb8b9464a8717a092c682c70c650e4"} Jul 22 12:41:44.040327 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:44.039972 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" event={"ID":"f033188f-9515-4e0f-96c3-d0bf016e6a46","Type":"ContainerStarted","Data":"00a5900f5c176eb62e1dd9ee3a61c77afa81f1463e679bdc33d656600bb80699"} Jul 22 12:41:44.060570 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:44.060446 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-vdn7r" podStartSLOduration=16.532592622 podStartE2EDuration="18.060429012s" totalImagesPullingTime="1.52783639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:44.059845298 +0000 UTC m=+57.864562195" watchObservedRunningTime="2026-07-22 12:41:44.060429012 +0000 UTC m=+57.865145904" Jul 22 12:41:46.932433 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:46.932406 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cdpvh" Jul 22 12:41:49.494936 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:49.494902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:49.497147 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:49.497127 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ef7f534d-ea89-4086-8ad1-d7a303cc8ec8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rb9sd\" (UID: \"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8\") " pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:49.611469 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:49.611442 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mvgjr\"" Jul 22 12:41:49.619611 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:49.619591 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rb9sd" Jul 22 12:41:49.707675 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:49.707656 2576 scope.go:117] "RemoveContainer" containerID="df7eaea6d64ec8d8e0ecb41a23989f6b599bc987bda028fcd10c5fd8d06419b6" Jul 22 12:41:49.752975 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:49.752952 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rb9sd"] Jul 22 12:41:49.755916 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:49.755894 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7f534d_ea89_4086_8ad1_d7a303cc8ec8.slice/crio-ebfca61959e5504b9842540f257f182f851fb300e8bcf3c44ea281ff62873f12 WatchSource:0}: Error finding container ebfca61959e5504b9842540f257f182f851fb300e8bcf3c44ea281ff62873f12: Status 404 returned error can't find the container with id ebfca61959e5504b9842540f257f182f851fb300e8bcf3c44ea281ff62873f12 Jul 22 12:41:50.058180 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:50.058109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:41:50.058340 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:50.058214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" event={"ID":"0550d49e-243e-437e-b4a6-5badb236b4fe","Type":"ContainerStarted","Data":"f5bb9979c55a8cce92eb8a5680c3bcda9833bcad44056292c38df637e66b4071"} Jul 22 12:41:50.058572 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:50.058543 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:50.059434 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:50.059407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rb9sd" event={"ID":"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8","Type":"ContainerStarted","Data":"ebfca61959e5504b9842540f257f182f851fb300e8bcf3c44ea281ff62873f12"} Jul 22 12:41:50.078093 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:50.078052 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" podStartSLOduration=18.155448688 podStartE2EDuration="24.078039466s" totalImagesPullingTime="5.922590778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:50.077047234 +0000 UTC m=+63.881764129" watchObservedRunningTime="2026-07-22 12:41:50.078039466 +0000 UTC m=+63.882756362" Jul 22 12:41:50.314973 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:50.314861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-rck2w" Jul 22 12:41:51.413069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.413023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:51.415722 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.415700 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 12:41:51.426348 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.426311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8f286d0-2bca-4211-9499-d80776bd4d35-metrics-certs\") pod \"network-metrics-daemon-tn8h2\" (UID: \"d8f286d0-2bca-4211-9499-d80776bd4d35\") " pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:51.513940 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.513902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:51.514099 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.513960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:51.516698 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.516673 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 12:41:51.516808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.516733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh46m\" (UniqueName: \"kubernetes.io/projected/ceea1b5d-f802-46e1-8391-111cfa88ed6d-kube-api-access-zh46m\") pod \"network-check-target-6wx8p\" (UID: \"ceea1b5d-f802-46e1-8391-111cfa88ed6d\") " pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:51.527196 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.527174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e10dae84-5ee9-4c3c-a86c-608de7b8504e-original-pull-secret\") pod \"global-pull-secret-syncer-4hvsk\" (UID: \"e10dae84-5ee9-4c3c-a86c-608de7b8504e\") " pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:51.722459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.722379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-72nf2\"" Jul 22 12:41:51.728198 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.728174 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv8qm\"" Jul 22 12:41:51.730107 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.730087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tn8h2" Jul 22 12:41:51.735984 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.735961 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:51.816628 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.816603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4hvsk" Jul 22 12:41:51.878727 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.878702 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tn8h2"] Jul 22 12:41:51.882497 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:51.882466 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f286d0_2bca_4211_9499_d80776bd4d35.slice/crio-76ded03d8e10d0daa945781795d0f7a915ff02224d7a693d61a77acb7d76545a WatchSource:0}: Error finding container 76ded03d8e10d0daa945781795d0f7a915ff02224d7a693d61a77acb7d76545a: Status 404 returned error can't find the container with id 76ded03d8e10d0daa945781795d0f7a915ff02224d7a693d61a77acb7d76545a Jul 22 12:41:51.895904 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.895814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6wx8p"] Jul 22 12:41:51.897818 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:51.897770 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceea1b5d_f802_46e1_8391_111cfa88ed6d.slice/crio-ebdc37497436fb918c5a45f897e8b7f1de2fbe311dab9b64df12f3dc7af44949 WatchSource:0}: Error finding container ebdc37497436fb918c5a45f897e8b7f1de2fbe311dab9b64df12f3dc7af44949: Status 404 returned error can't find the container with id ebdc37497436fb918c5a45f897e8b7f1de2fbe311dab9b64df12f3dc7af44949 Jul 22 12:41:51.968005 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:51.967980 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4hvsk"] Jul 22 12:41:52.068220 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:52.068146 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6wx8p" event={"ID":"ceea1b5d-f802-46e1-8391-111cfa88ed6d","Type":"ContainerStarted","Data":"e75d90865f76489acc6e3a565421e825febd09d1c5c7734b66b0eb12f559f091"} Jul 22 12:41:52.068220 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:52.068207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6wx8p" event={"ID":"ceea1b5d-f802-46e1-8391-111cfa88ed6d","Type":"ContainerStarted","Data":"ebdc37497436fb918c5a45f897e8b7f1de2fbe311dab9b64df12f3dc7af44949"} Jul 22 12:41:52.068435 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:52.068262 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:41:52.069509 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:52.069485 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tn8h2" event={"ID":"d8f286d0-2bca-4211-9499-d80776bd4d35","Type":"ContainerStarted","Data":"76ded03d8e10d0daa945781795d0f7a915ff02224d7a693d61a77acb7d76545a"} Jul 22 12:41:52.086331 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:52.086290 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6wx8p" podStartSLOduration=66.086275932 podStartE2EDuration="1m6.086275932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:52.085989337 +0000 UTC m=+65.890706234" watchObservedRunningTime="2026-07-22 12:41:52.086275932 +0000 UTC m=+65.890992829" Jul 22 12:41:53.025906 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:53.025872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:53.025906 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:53.025913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:53.028622 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:53.028595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72cd096f-6bab-4c50-9088-186c56d2a701-metrics-tls\") pod \"dns-default-qkvwk\" (UID: \"72cd096f-6bab-4c50-9088-186c56d2a701\") " pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:53.028802 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:53.028649 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"image-registry-57b85889f4-sqqhz\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:53.224732 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:53.224706 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fgszx\"" Jul 22 12:41:53.233110 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:53.233076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:53.247877 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:53.247848 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kbrpx\"" Jul 22 12:41:53.256151 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:53.256130 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:54.056194 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:54.056166 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10dae84_5ee9_4c3c_a86c_608de7b8504e.slice/crio-209ee5e39792aaebc282c91d2249cdcf69fb653ad4857341914db19713dde272 WatchSource:0}: Error finding container 209ee5e39792aaebc282c91d2249cdcf69fb653ad4857341914db19713dde272: Status 404 returned error can't find the container with id 209ee5e39792aaebc282c91d2249cdcf69fb653ad4857341914db19713dde272 Jul 22 12:41:54.077389 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:54.077362 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4hvsk" event={"ID":"e10dae84-5ee9-4c3c-a86c-608de7b8504e","Type":"ContainerStarted","Data":"209ee5e39792aaebc282c91d2249cdcf69fb653ad4857341914db19713dde272"} Jul 22 12:41:55.156872 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:55.156840 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b85889f4-sqqhz"] Jul 22 12:41:55.159503 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:55.159476 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fb5022_5aa2_4857_b737_a8d9ab7b6ab4.slice/crio-210bd2c6b23065423034307c58f7f2e54fec914551836e4f3a53aae4680ef119 WatchSource:0}: Error finding container 210bd2c6b23065423034307c58f7f2e54fec914551836e4f3a53aae4680ef119: Status 404 returned error can't find the container with id 210bd2c6b23065423034307c58f7f2e54fec914551836e4f3a53aae4680ef119 Jul 22 12:41:55.169348 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:55.169325 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qkvwk"] Jul 22 12:41:55.171762 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:41:55.171739 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cd096f_6bab_4c50_9088_186c56d2a701.slice/crio-48691a0a6a3bc06be4bb911271a76eafc3ab2c53dfe3f380bc96e8eb49abd34d WatchSource:0}: Error finding container 48691a0a6a3bc06be4bb911271a76eafc3ab2c53dfe3f380bc96e8eb49abd34d: Status 404 returned error can't find the container with id 48691a0a6a3bc06be4bb911271a76eafc3ab2c53dfe3f380bc96e8eb49abd34d Jul 22 12:41:56.089990 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.089803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rb9sd" event={"ID":"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8","Type":"ContainerStarted","Data":"0a8cf0b6eba685cc93b0a9ae8a3df7d6dc4ec408fcfd11185f1cc24f8db627dc"} Jul 22 12:41:56.091207 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.091171 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkvwk" event={"ID":"72cd096f-6bab-4c50-9088-186c56d2a701","Type":"ContainerStarted","Data":"48691a0a6a3bc06be4bb911271a76eafc3ab2c53dfe3f380bc96e8eb49abd34d"} Jul 22 12:41:56.093691 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.093631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tn8h2" event={"ID":"d8f286d0-2bca-4211-9499-d80776bd4d35","Type":"ContainerStarted","Data":"b16882901f5d3c96a85136d20723281fc02bd4b4a58ea0039389557cf4d6b7c2"} Jul 22 12:41:56.093691 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.093682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tn8h2" event={"ID":"d8f286d0-2bca-4211-9499-d80776bd4d35","Type":"ContainerStarted","Data":"05edc3e4d7ded345f6df0d55212c508d8652fa2062b641c32308cc8ae678d186"} Jul 22 12:41:56.095381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.095356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" event={"ID":"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4","Type":"ContainerStarted","Data":"4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7"} Jul 22 12:41:56.095472 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.095394 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" event={"ID":"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4","Type":"ContainerStarted","Data":"210bd2c6b23065423034307c58f7f2e54fec914551836e4f3a53aae4680ef119"} Jul 22 12:41:56.095610 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.095586 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:41:56.111403 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.111357 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tn8h2" podStartSLOduration=66.185398193 podStartE2EDuration="1m10.111343477s" totalImagesPullingTime="3.925945284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:56.11038479 +0000 UTC m=+69.915101686" watchObservedRunningTime="2026-07-22 12:41:56.111343477 +0000 UTC m=+69.916060372" Jul 22 12:41:56.131049 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:56.130992 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" podStartSLOduration=69.13097823 podStartE2EDuration="1m9.13097823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:56.129502191 +0000 UTC m=+69.934219087" watchObservedRunningTime="2026-07-22 12:41:56.13097823 +0000 UTC m=+69.935695124" Jul 22 12:41:57.975722 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:57.975687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:57.976192 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:57.975734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:57.976386 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:57.976358 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-service-ca-bundle\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:57.978382 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:57.978363 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7a2923-60f8-434a-9cc3-2cc73825ddcf-metrics-certs\") pod \"router-default-c964bf7f6-tqffj\" (UID: \"5e7a2923-60f8-434a-9cc3-2cc73825ddcf\") " pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:58.105264 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.104950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkvwk" event={"ID":"72cd096f-6bab-4c50-9088-186c56d2a701","Type":"ContainerStarted","Data":"db6725442bd1508a4610e32ec9533514cd4458a6dde25a47c1c95eb364fd06e2"} Jul 22 12:41:58.105264 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.105031 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qkvwk" event={"ID":"72cd096f-6bab-4c50-9088-186c56d2a701","Type":"ContainerStarted","Data":"44837153171c07bedb2bb956f63f06a8e441180ce1ced1498fad44041c28efa4"} Jul 22 12:41:58.105264 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.105106 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qkvwk" Jul 22 12:41:58.125955 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.125915 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qkvwk" podStartSLOduration=34.87240733 podStartE2EDuration="37.125901458s" totalImagesPullingTime="2.253494128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:58.125251619 +0000 UTC m=+71.929968516" watchObservedRunningTime="2026-07-22 12:41:58.125901458 +0000 UTC m=+71.930618353" Jul 22 12:41:58.172301 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.172276 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-m85m6\"" Jul 22 12:41:58.177869 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.177847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:58.180117 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.180096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:41:58.180454 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.180436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f68990c1-be9f-4a0a-9165-cb17dcfa2ec3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-8fq6j\" (UID: \"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:41:58.309164 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.309093 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q4jdp\"" Jul 22 12:41:58.316522 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:41:58.316498 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" Jul 22 12:42:00.980993 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:00.980966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j"] Jul 22 12:42:00.984070 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:42:00.984004 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf68990c1_be9f_4a0a_9165_cb17dcfa2ec3.slice/crio-5b2d8caea1daf87efaf46c54d38d34fc9869b1f55ee08a3318dfab076277ee85 WatchSource:0}: Error finding container 5b2d8caea1daf87efaf46c54d38d34fc9869b1f55ee08a3318dfab076277ee85: Status 404 returned error can't find the container with id 5b2d8caea1daf87efaf46c54d38d34fc9869b1f55ee08a3318dfab076277ee85 Jul 22 12:42:00.998059 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:00.997718 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c964bf7f6-tqffj"] Jul 22 12:42:01.002794 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:42:01.002769 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7a2923_60f8_434a_9cc3_2cc73825ddcf.slice/crio-099bf051868f6bbe40bc02fa95c982834c493c8714cce517b14aada78a423b7f WatchSource:0}: Error finding container 099bf051868f6bbe40bc02fa95c982834c493c8714cce517b14aada78a423b7f: Status 404 returned error can't find the container with id 099bf051868f6bbe40bc02fa95c982834c493c8714cce517b14aada78a423b7f Jul 22 12:42:01.118470 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.118392 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rb9sd" event={"ID":"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8","Type":"ContainerStarted","Data":"03e3bf236207f4f06fae81f5f14761ce05fbe08903cd75dd62812adfbbf192ca"} Jul 22 12:42:01.119925 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.119900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4hvsk" event={"ID":"e10dae84-5ee9-4c3c-a86c-608de7b8504e","Type":"ContainerStarted","Data":"5acca990e60dbdc111a049242a8021fa7084e2d38f78ae77111b69b6ec962bb0"} Jul 22 12:42:01.121162 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.121131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" event={"ID":"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3","Type":"ContainerStarted","Data":"5b2d8caea1daf87efaf46c54d38d34fc9869b1f55ee08a3318dfab076277ee85"} Jul 22 12:42:01.122561 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.122540 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c964bf7f6-tqffj" event={"ID":"5e7a2923-60f8-434a-9cc3-2cc73825ddcf","Type":"ContainerStarted","Data":"af6c49d87d3cb13030735655b0b81006b78e605631e9ec64bda8af09e4b17340"} Jul 22 12:42:01.122641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.122574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c964bf7f6-tqffj" event={"ID":"5e7a2923-60f8-434a-9cc3-2cc73825ddcf","Type":"ContainerStarted","Data":"099bf051868f6bbe40bc02fa95c982834c493c8714cce517b14aada78a423b7f"} Jul 22 12:42:01.137446 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.137408 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4hvsk" podStartSLOduration=67.340758828 podStartE2EDuration="1m14.137396066s" totalImagesPullingTime="6.796637238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:01.136540204 +0000 UTC m=+74.941257101" watchObservedRunningTime="2026-07-22 12:42:01.137396066 +0000 UTC m=+74.942112961" Jul 22 12:42:01.155968 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.155925 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c964bf7f6-tqffj" podStartSLOduration=36.155912755 podStartE2EDuration="36.155912755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:01.155326008 +0000 UTC m=+74.960042908" watchObservedRunningTime="2026-07-22 12:42:01.155912755 +0000 UTC m=+74.960629650" Jul 22 12:42:01.180790 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.180769 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:42:01.183154 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.183125 2576 patch_prober.go:28] interesting pod/router-default-c964bf7f6-tqffj container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld Jul 22 12:42:01.183154 ip-10-0-137-88 kubenswrapper[2576]: [-]has-synced failed: reason withheld Jul 22 12:42:01.183154 ip-10-0-137-88 kubenswrapper[2576]: [+]process-running ok Jul 22 12:42:01.183154 ip-10-0-137-88 kubenswrapper[2576]: healthz check failed Jul 22 12:42:01.183330 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:01.183175 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-c964bf7f6-tqffj" podUID="5e7a2923-60f8-434a-9cc3-2cc73825ddcf" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 22 12:42:02.182359 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:02.182328 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:42:03.131198 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:03.131134 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:42:03.132449 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:03.132428 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c964bf7f6-tqffj" Jul 22 12:42:06.142143 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:06.142109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" event={"ID":"f68990c1-be9f-4a0a-9165-cb17dcfa2ec3","Type":"ContainerStarted","Data":"6501579c5f20cad29d2f07b7fde7a61d012f3a4d29e1828e48d9ae1b09f5801c"} Jul 22 12:42:06.144152 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:06.144129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rb9sd" event={"ID":"ef7f534d-ea89-4086-8ad1-d7a303cc8ec8","Type":"ContainerStarted","Data":"d76527ee4efe425f81a9f93007d57d3a12a62ea1ca056cd0e6fc058f03eebc20"} Jul 22 12:42:06.172500 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:06.172458 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-8fq6j" podStartSLOduration=36.007784887 podStartE2EDuration="40.172442958s" totalImagesPullingTime="4.164658071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:06.172254075 +0000 UTC m=+79.976970981" watchObservedRunningTime="2026-07-22 12:42:06.172442958 +0000 UTC m=+79.977159873" Jul 22 12:42:06.201994 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:06.201953 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rb9sd" podStartSLOduration=17.986769371 podStartE2EDuration="33.201939297s" totalImagesPullingTime="15.215169926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:06.201511131 +0000 UTC m=+80.006228045" watchObservedRunningTime="2026-07-22 12:42:06.201939297 +0000 UTC m=+80.006656193" Jul 22 12:42:08.111490 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:08.111387 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qkvwk" Jul 22 12:42:13.200519 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.200487 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f7d5q"] Jul 22 12:42:13.203389 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.203374 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.205739 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.205713 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2xzrv\"" Jul 22 12:42:13.206347 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.206328 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 12:42:13.206421 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.206330 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 12:42:13.206613 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.206599 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 12:42:13.206688 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.206678 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 12:42:13.291270 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-sys\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.291416 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-tls\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.291416 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.291416 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291382 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-root\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.291578 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d32b678c-ced0-4f12-ad48-02f6dde24b37-metrics-client-ca\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.291578 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-accelerators-collector-config\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.291578 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5fl\" (UniqueName: \"kubernetes.io/projected/d32b678c-ced0-4f12-ad48-02f6dde24b37-kube-api-access-js5fl\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.291578 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291519 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-wtmp\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.291578 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.291544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-textfile\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392148 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392122 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-wtmp\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-textfile\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-sys\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-tls\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392281 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-root\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-sys\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d32b678c-ced0-4f12-ad48-02f6dde24b37-metrics-client-ca\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392345 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-wtmp\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392668 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-accelerators-collector-config\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392668 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5fl\" (UniqueName: \"kubernetes.io/projected/d32b678c-ced0-4f12-ad48-02f6dde24b37-kube-api-access-js5fl\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392668 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:42:13.392392 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 12:42:13.392668 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d32b678c-ced0-4f12-ad48-02f6dde24b37-root\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392668 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:42:13.392469 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-tls podName:d32b678c-ced0-4f12-ad48-02f6dde24b37 nodeName:}" failed. No retries permitted until 2026-07-22 12:42:13.892448617 +0000 UTC m=+87.697165495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-tls") pod "node-exporter-f7d5q" (UID: "d32b678c-ced0-4f12-ad48-02f6dde24b37") : secret "node-exporter-tls" not found Jul 22 12:42:13.392668 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-textfile\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392904 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392839 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d32b678c-ced0-4f12-ad48-02f6dde24b37-metrics-client-ca\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.392904 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.392851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-accelerators-collector-config\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.394649 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.394633 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.403623 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.403598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5fl\" (UniqueName: \"kubernetes.io/projected/d32b678c-ced0-4f12-ad48-02f6dde24b37-kube-api-access-js5fl\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.896136 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.896102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-tls\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:13.898456 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:13.898431 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d32b678c-ced0-4f12-ad48-02f6dde24b37-node-exporter-tls\") pod \"node-exporter-f7d5q\" (UID: \"d32b678c-ced0-4f12-ad48-02f6dde24b37\") " pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:14.112398 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:14.112367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f7d5q" Jul 22 12:42:14.123171 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:42:14.123146 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32b678c_ced0_4f12_ad48_02f6dde24b37.slice/crio-78f398c39c5fefd8ac5ad44b3f981ce8aae769b540b6ec78dceab114febea494 WatchSource:0}: Error finding container 78f398c39c5fefd8ac5ad44b3f981ce8aae769b540b6ec78dceab114febea494: Status 404 returned error can't find the container with id 78f398c39c5fefd8ac5ad44b3f981ce8aae769b540b6ec78dceab114febea494 Jul 22 12:42:14.171741 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:14.171712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f7d5q" event={"ID":"d32b678c-ced0-4f12-ad48-02f6dde24b37","Type":"ContainerStarted","Data":"78f398c39c5fefd8ac5ad44b3f981ce8aae769b540b6ec78dceab114febea494"} Jul 22 12:42:15.177222 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:15.177192 2576 generic.go:356] "Generic (PLEG): container finished" podID="d32b678c-ced0-4f12-ad48-02f6dde24b37" containerID="c2fb3982cb510c34eecd4a6f09eafe3a29eac56029c8626134849c1c6a852055" exitCode=0 Jul 22 12:42:15.177579 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:15.177288 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f7d5q" event={"ID":"d32b678c-ced0-4f12-ad48-02f6dde24b37","Type":"ContainerDied","Data":"c2fb3982cb510c34eecd4a6f09eafe3a29eac56029c8626134849c1c6a852055"} Jul 22 12:42:16.182416 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:16.182387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f7d5q" event={"ID":"d32b678c-ced0-4f12-ad48-02f6dde24b37","Type":"ContainerStarted","Data":"df96572aed27e476f2d4eeb79709908d2a94691da83be250f9929671dfbf7de1"} Jul 22 12:42:16.182776 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:16.182437 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f7d5q" event={"ID":"d32b678c-ced0-4f12-ad48-02f6dde24b37","Type":"ContainerStarted","Data":"9a0dfd84f4c3dfa19eaa5ca53ab12c9347b1fb543df812bd7d327a1b5b33f7c0"} Jul 22 12:42:16.206244 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:16.206198 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f7d5q" podStartSLOduration=2.309622213 podStartE2EDuration="3.206179407s" totalImagesPullingTime="824.018622ms" totalInitContainerRuntime="72.538572ms" isStatefulPod=true podCreationTimestamp="2026-07-22 12:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:16.204915656 +0000 UTC m=+90.009632578" watchObservedRunningTime="2026-07-22 12:42:16.206179407 +0000 UTC m=+90.010896304" Jul 22 12:42:17.103781 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:17.103756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:42:22.842185 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:22.842150 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b85889f4-sqqhz"] Jul 22 12:42:23.076483 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:23.076449 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6wx8p" Jul 22 12:42:47.862335 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:47.862295 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" podUID="e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" containerName="registry" containerID="cri-o://4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7" gracePeriod=30 Jul 22 12:42:48.093511 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.093489 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:42:48.146407 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146351 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-certificates\") pod \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " Jul 22 12:42:48.146407 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146393 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") pod \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " Jul 22 12:42:48.146587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146422 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-installation-pull-secrets\") pod \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " Jul 22 12:42:48.146587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146443 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-ca-trust-extracted\") pod \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " Jul 22 12:42:48.146587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-bound-sa-token\") pod \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " Jul 22 12:42:48.146587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxhzj\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-kube-api-access-hxhzj\") pod \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " Jul 22 12:42:48.146587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146537 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-trusted-ca\") pod \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " Jul 22 12:42:48.146587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-image-registry-private-configuration\") pod \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\" (UID: \"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4\") " Jul 22 12:42:48.146932 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146906 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:48.147026 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.146985 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:42:48.149092 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.149041 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:42:48.149497 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.149452 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:42:48.149597 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.149504 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:42:48.149658 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.149616 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-kube-api-access-hxhzj" (OuterVolumeSpecName: "kube-api-access-hxhzj") pod "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4"). InnerVolumeSpecName "kube-api-access-hxhzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:42:48.149756 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.149734 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:42:48.156239 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.156217 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" (UID: "e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:42:48.247711 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.247690 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-installation-pull-secrets\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.247808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.247717 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-ca-trust-extracted\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.247808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.247730 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-bound-sa-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.247808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.247739 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxhzj\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-kube-api-access-hxhzj\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.247808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.247747 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-trusted-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.247808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.247776 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-image-registry-private-configuration\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.247808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.247785 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-certificates\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.247808 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.247793 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4-registry-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:42:48.297449 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.297427 2576 generic.go:356] "Generic (PLEG): container finished" podID="e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" containerID="4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7" exitCode=0 Jul 22 12:42:48.297528 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.297459 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" event={"ID":"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4","Type":"ContainerDied","Data":"4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7"} Jul 22 12:42:48.297528 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.297473 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" Jul 22 12:42:48.297528 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.297497 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b85889f4-sqqhz" event={"ID":"e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4","Type":"ContainerDied","Data":"210bd2c6b23065423034307c58f7f2e54fec914551836e4f3a53aae4680ef119"} Jul 22 12:42:48.297528 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.297517 2576 scope.go:117] "RemoveContainer" containerID="4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7" Jul 22 12:42:48.329614 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.329595 2576 scope.go:117] "RemoveContainer" containerID="4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7" Jul 22 12:42:48.329895 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:42:48.329876 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7\": container with ID starting with 4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7 not found: ID does not exist" containerID="4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7" Jul 22 12:42:48.329948 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.329904 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7"} err="failed to get container status \"4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7\": rpc error: code = NotFound desc = could not find container \"4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7\": container with ID starting with 4dc3baca41c4df8c9df56bd191dec88e5e555a816b06436caf01b8cfcddc58a7 not found: ID does not exist" Jul 22 12:42:48.341098 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.341076 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b85889f4-sqqhz"] Jul 22 12:42:48.347988 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.347962 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57b85889f4-sqqhz"] Jul 22 12:42:48.711914 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:48.711879 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" path="/var/lib/kubelet/pods/e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4/volumes" Jul 22 12:42:49.302845 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:49.302815 2576 generic.go:356] "Generic (PLEG): container finished" podID="0701213e-6c4b-41aa-aac6-38e0f6cc47f9" containerID="38896578e0e92e15de4a4d58ced7a52acbbf37cf7d7970580b251fbfec94bc51" exitCode=0 Jul 22 12:42:49.303240 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:49.302887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" event={"ID":"0701213e-6c4b-41aa-aac6-38e0f6cc47f9","Type":"ContainerDied","Data":"38896578e0e92e15de4a4d58ced7a52acbbf37cf7d7970580b251fbfec94bc51"} Jul 22 12:42:49.303332 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:49.303245 2576 scope.go:117] "RemoveContainer" containerID="38896578e0e92e15de4a4d58ced7a52acbbf37cf7d7970580b251fbfec94bc51" Jul 22 12:42:49.304408 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:49.304318 2576 generic.go:356] "Generic (PLEG): container finished" podID="475631b0-ff58-405b-bdca-7228773d131d" containerID="9c22b6efa943b1f46ea1c6ac113aec80feb90e2e77ca5ad6469ce8370b2fcf26" exitCode=0 Jul 22 12:42:49.304408 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:49.304369 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8d228" event={"ID":"475631b0-ff58-405b-bdca-7228773d131d","Type":"ContainerDied","Data":"9c22b6efa943b1f46ea1c6ac113aec80feb90e2e77ca5ad6469ce8370b2fcf26"} Jul 22 12:42:49.304685 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:49.304668 2576 scope.go:117] "RemoveContainer" containerID="9c22b6efa943b1f46ea1c6ac113aec80feb90e2e77ca5ad6469ce8370b2fcf26" Jul 22 12:42:50.310894 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:50.310861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-hgzzk" event={"ID":"0701213e-6c4b-41aa-aac6-38e0f6cc47f9","Type":"ContainerStarted","Data":"e3d73ab68ada62d9a352ec970a09e603ceb4e69e36625a1f091f85c6f5e4295c"} Jul 22 12:42:50.312616 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:50.312591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-8d228" event={"ID":"475631b0-ff58-405b-bdca-7228773d131d","Type":"ContainerStarted","Data":"ee70f2f6fe98f0f50b7a7751cdfa724ddfa3faddf3dc12216d591167cebc0a34"} Jul 22 12:42:59.346547 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:59.346511 2576 generic.go:356] "Generic (PLEG): container finished" podID="749ddc12-d2c3-4c26-aede-b69956acd623" containerID="0b999f80c1becb041a2798cf63061af97529077a8e5fd81e5d5c5b9abd375c62" exitCode=0 Jul 22 12:42:59.347033 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:59.346594 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" event={"ID":"749ddc12-d2c3-4c26-aede-b69956acd623","Type":"ContainerDied","Data":"0b999f80c1becb041a2798cf63061af97529077a8e5fd81e5d5c5b9abd375c62"} Jul 22 12:42:59.347033 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:42:59.346951 2576 scope.go:117] "RemoveContainer" containerID="0b999f80c1becb041a2798cf63061af97529077a8e5fd81e5d5c5b9abd375c62" Jul 22 12:43:00.352151 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:43:00.352121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-n2m54" event={"ID":"749ddc12-d2c3-4c26-aede-b69956acd623","Type":"ContainerStarted","Data":"d583d77d8e4fea569f1ed50ce9908b0f643a8633535509e8de83684aba3acf1d"} Jul 22 12:44:42.165539 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.165508 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97"] Jul 22 12:44:42.166005 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.165754 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" containerName="registry" Jul 22 12:44:42.166005 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.165765 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" containerName="registry" Jul 22 12:44:42.166005 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.165814 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6fb5022-5aa2-4857-b737-a8d9ab7b6ab4" containerName="registry" Jul 22 12:44:42.168806 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.168787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.172141 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.172117 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 22 12:44:42.172141 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.172130 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 12:44:42.172298 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.172127 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 12:44:42.172298 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.172119 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 12:44:42.179311 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.179289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97"] Jul 22 12:44:42.233365 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.233340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/159f91e6-eb8f-44f9-ac96-fd600c5619a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.233460 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.233369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmw6\" (UniqueName: \"kubernetes.io/projected/159f91e6-eb8f-44f9-ac96-fd600c5619a4-kube-api-access-czmw6\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.233460 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.233395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/159f91e6-eb8f-44f9-ac96-fd600c5619a4-tmp\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.334248 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.334225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/159f91e6-eb8f-44f9-ac96-fd600c5619a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.334334 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.334251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmw6\" (UniqueName: \"kubernetes.io/projected/159f91e6-eb8f-44f9-ac96-fd600c5619a4-kube-api-access-czmw6\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.334334 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.334276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/159f91e6-eb8f-44f9-ac96-fd600c5619a4-tmp\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.334565 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.334551 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/159f91e6-eb8f-44f9-ac96-fd600c5619a4-tmp\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.336823 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.336797 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/159f91e6-eb8f-44f9-ac96-fd600c5619a4-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.343002 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.342981 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmw6\" (UniqueName: \"kubernetes.io/projected/159f91e6-eb8f-44f9-ac96-fd600c5619a4-kube-api-access-czmw6\") pod \"klusterlet-addon-workmgr-6d79cb4657-txd97\" (UID: \"159f91e6-eb8f-44f9-ac96-fd600c5619a4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.478061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.477991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:42.596525 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.596498 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97"] Jul 22 12:44:42.599513 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:44:42.599487 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159f91e6_eb8f_44f9_ac96_fd600c5619a4.slice/crio-ec7c000ce8b475515ee546e48a4bd06e909665f777a1e2e875555fceafb75d2b WatchSource:0}: Error finding container ec7c000ce8b475515ee546e48a4bd06e909665f777a1e2e875555fceafb75d2b: Status 404 returned error can't find the container with id ec7c000ce8b475515ee546e48a4bd06e909665f777a1e2e875555fceafb75d2b Jul 22 12:44:42.710757 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:42.710734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" event={"ID":"159f91e6-eb8f-44f9-ac96-fd600c5619a4","Type":"ContainerStarted","Data":"ec7c000ce8b475515ee546e48a4bd06e909665f777a1e2e875555fceafb75d2b"} Jul 22 12:44:46.733784 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:46.733746 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" event={"ID":"159f91e6-eb8f-44f9-ac96-fd600c5619a4","Type":"ContainerStarted","Data":"a4580f96df2bc2b5b4965c780b3fab16b234a447aab0f5fa05d3e39813ce1302"} Jul 22 12:44:46.734205 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:46.733920 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:46.735763 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:46.735741 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" Jul 22 12:44:46.750139 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:44:46.750098 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d79cb4657-txd97" podStartSLOduration=1.2669868 podStartE2EDuration="4.750084954s" totalImagesPullingTime="3.483098154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:44:46.748495642 +0000 UTC m=+240.553212537" watchObservedRunningTime="2026-07-22 12:44:46.750084954 +0000 UTC m=+240.554801846" Jul 22 12:45:13.438254 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.438177 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx"] Jul 22 12:45:13.440211 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.440196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.442621 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.442595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 12:45:13.442737 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.442644 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f8jfx\"" Jul 22 12:45:13.442737 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.442684 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 12:45:13.450065 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.450043 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx"] Jul 22 12:45:13.549111 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.549087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b5p2\" (UniqueName: \"kubernetes.io/projected/4a367253-1697-4f10-9e38-91d463cfba85-kube-api-access-9b5p2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.549208 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.549141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.549250 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.549205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.649874 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.649851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.649985 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.649887 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.649985 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.649916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b5p2\" (UniqueName: \"kubernetes.io/projected/4a367253-1697-4f10-9e38-91d463cfba85-kube-api-access-9b5p2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.650213 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.650194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.650279 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.650246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.658778 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.658749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b5p2\" (UniqueName: \"kubernetes.io/projected/4a367253-1697-4f10-9e38-91d463cfba85-kube-api-access-9b5p2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.749370 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.749311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:13.869944 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:13.869843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx"] Jul 22 12:45:13.872690 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:45:13.872665 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a367253_1697_4f10_9e38_91d463cfba85.slice/crio-efcefdd7d6ad2c95dd0d95d5d2290ad4d12eb6ae5daa82d74fb1c34281b763cc WatchSource:0}: Error finding container efcefdd7d6ad2c95dd0d95d5d2290ad4d12eb6ae5daa82d74fb1c34281b763cc: Status 404 returned error can't find the container with id efcefdd7d6ad2c95dd0d95d5d2290ad4d12eb6ae5daa82d74fb1c34281b763cc Jul 22 12:45:14.830107 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:14.828656 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" event={"ID":"4a367253-1697-4f10-9e38-91d463cfba85","Type":"ContainerStarted","Data":"efcefdd7d6ad2c95dd0d95d5d2290ad4d12eb6ae5daa82d74fb1c34281b763cc"} Jul 22 12:45:20.853652 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:20.853615 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a367253-1697-4f10-9e38-91d463cfba85" containerID="a6658f95f9a77a5a71d44a3b2aaf5849559ea3ac213b2463b2b0343d57352fd1" exitCode=0 Jul 22 12:45:20.854050 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:20.853689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" event={"ID":"4a367253-1697-4f10-9e38-91d463cfba85","Type":"ContainerDied","Data":"a6658f95f9a77a5a71d44a3b2aaf5849559ea3ac213b2463b2b0343d57352fd1"} Jul 22 12:45:23.867075 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:23.867034 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a367253-1697-4f10-9e38-91d463cfba85" containerID="ce0709e46a77ae873b002e45110453dad2f2fde1c7c5d3fcfe1b4ede9792e176" exitCode=0 Jul 22 12:45:23.867527 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:23.867099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" event={"ID":"4a367253-1697-4f10-9e38-91d463cfba85","Type":"ContainerDied","Data":"ce0709e46a77ae873b002e45110453dad2f2fde1c7c5d3fcfe1b4ede9792e176"} Jul 22 12:45:30.899618 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:30.899531 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a367253-1697-4f10-9e38-91d463cfba85" containerID="29455cfd479e21d3944c053c2fa1d1d9b7c055b15a0ff48aadf2a9680050f40b" exitCode=0 Jul 22 12:45:30.899958 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:30.899618 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" event={"ID":"4a367253-1697-4f10-9e38-91d463cfba85","Type":"ContainerDied","Data":"29455cfd479e21d3944c053c2fa1d1d9b7c055b15a0ff48aadf2a9680050f40b"} Jul 22 12:45:32.028380 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.028359 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:32.091530 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.091490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-bundle\") pod \"4a367253-1697-4f10-9e38-91d463cfba85\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " Jul 22 12:45:32.091648 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.091543 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-util\") pod \"4a367253-1697-4f10-9e38-91d463cfba85\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " Jul 22 12:45:32.091648 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.091565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b5p2\" (UniqueName: \"kubernetes.io/projected/4a367253-1697-4f10-9e38-91d463cfba85-kube-api-access-9b5p2\") pod \"4a367253-1697-4f10-9e38-91d463cfba85\" (UID: \"4a367253-1697-4f10-9e38-91d463cfba85\") " Jul 22 12:45:32.092268 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.092239 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-bundle" (OuterVolumeSpecName: "bundle") pod "4a367253-1697-4f10-9e38-91d463cfba85" (UID: "4a367253-1697-4f10-9e38-91d463cfba85"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:45:32.093727 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.093705 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a367253-1697-4f10-9e38-91d463cfba85-kube-api-access-9b5p2" (OuterVolumeSpecName: "kube-api-access-9b5p2") pod "4a367253-1697-4f10-9e38-91d463cfba85" (UID: "4a367253-1697-4f10-9e38-91d463cfba85"). InnerVolumeSpecName "kube-api-access-9b5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:45:32.095840 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.095822 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-util" (OuterVolumeSpecName: "util") pod "4a367253-1697-4f10-9e38-91d463cfba85" (UID: "4a367253-1697-4f10-9e38-91d463cfba85"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:45:32.192906 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.192870 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-bundle\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:45:32.192906 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.192904 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a367253-1697-4f10-9e38-91d463cfba85-util\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:45:32.192906 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.192914 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b5p2\" (UniqueName: \"kubernetes.io/projected/4a367253-1697-4f10-9e38-91d463cfba85-kube-api-access-9b5p2\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:45:32.907563 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.907534 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" Jul 22 12:45:32.907563 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.907555 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540knxdx" event={"ID":"4a367253-1697-4f10-9e38-91d463cfba85","Type":"ContainerDied","Data":"efcefdd7d6ad2c95dd0d95d5d2290ad4d12eb6ae5daa82d74fb1c34281b763cc"} Jul 22 12:45:32.907758 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:32.907602 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efcefdd7d6ad2c95dd0d95d5d2290ad4d12eb6ae5daa82d74fb1c34281b763cc" Jul 22 12:45:36.460406 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.460369 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc"] Jul 22 12:45:36.460898 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.460631 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a367253-1697-4f10-9e38-91d463cfba85" containerName="extract" Jul 22 12:45:36.460898 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.460640 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a367253-1697-4f10-9e38-91d463cfba85" containerName="extract" Jul 22 12:45:36.460898 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.460655 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a367253-1697-4f10-9e38-91d463cfba85" containerName="util" Jul 22 12:45:36.460898 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.460660 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a367253-1697-4f10-9e38-91d463cfba85" containerName="util" Jul 22 12:45:36.460898 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.460666 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a367253-1697-4f10-9e38-91d463cfba85" containerName="pull" Jul 22 12:45:36.460898 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.460670 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a367253-1697-4f10-9e38-91d463cfba85" containerName="pull" Jul 22 12:45:36.460898 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.460712 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a367253-1697-4f10-9e38-91d463cfba85" containerName="extract" Jul 22 12:45:36.505458 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.505430 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc"] Jul 22 12:45:36.505592 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.505514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" Jul 22 12:45:36.508430 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.508401 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 22 12:45:36.508430 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.508421 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:45:36.508579 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.508401 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mrcll\"" Jul 22 12:45:36.631282 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.631257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xr4n\" (UniqueName: \"kubernetes.io/projected/8426be7e-d9b0-443f-8ed6-3e36d4f258ce-kube-api-access-8xr4n\") pod \"cert-manager-operator-controller-manager-55c444bf86-qw8xc\" (UID: \"8426be7e-d9b0-443f-8ed6-3e36d4f258ce\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" Jul 22 12:45:36.631398 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.631304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8426be7e-d9b0-443f-8ed6-3e36d4f258ce-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qw8xc\" (UID: \"8426be7e-d9b0-443f-8ed6-3e36d4f258ce\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" Jul 22 12:45:36.732205 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.732140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8426be7e-d9b0-443f-8ed6-3e36d4f258ce-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qw8xc\" (UID: \"8426be7e-d9b0-443f-8ed6-3e36d4f258ce\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" Jul 22 12:45:36.732330 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.732224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xr4n\" (UniqueName: \"kubernetes.io/projected/8426be7e-d9b0-443f-8ed6-3e36d4f258ce-kube-api-access-8xr4n\") pod \"cert-manager-operator-controller-manager-55c444bf86-qw8xc\" (UID: \"8426be7e-d9b0-443f-8ed6-3e36d4f258ce\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" Jul 22 12:45:36.732551 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.732533 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8426be7e-d9b0-443f-8ed6-3e36d4f258ce-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-qw8xc\" (UID: \"8426be7e-d9b0-443f-8ed6-3e36d4f258ce\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" Jul 22 12:45:36.740780 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.740758 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xr4n\" (UniqueName: \"kubernetes.io/projected/8426be7e-d9b0-443f-8ed6-3e36d4f258ce-kube-api-access-8xr4n\") pod \"cert-manager-operator-controller-manager-55c444bf86-qw8xc\" (UID: \"8426be7e-d9b0-443f-8ed6-3e36d4f258ce\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" Jul 22 12:45:36.813460 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.813438 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" Jul 22 12:45:36.946440 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:36.946350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc"] Jul 22 12:45:36.948808 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:45:36.948779 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8426be7e_d9b0_443f_8ed6_3e36d4f258ce.slice/crio-419db8e4e8d1971c02000b894934820c594214d8009b1d348bf4e2d20ce30f7d WatchSource:0}: Error finding container 419db8e4e8d1971c02000b894934820c594214d8009b1d348bf4e2d20ce30f7d: Status 404 returned error can't find the container with id 419db8e4e8d1971c02000b894934820c594214d8009b1d348bf4e2d20ce30f7d Jul 22 12:45:37.927074 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:37.927043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" event={"ID":"8426be7e-d9b0-443f-8ed6-3e36d4f258ce","Type":"ContainerStarted","Data":"419db8e4e8d1971c02000b894934820c594214d8009b1d348bf4e2d20ce30f7d"} Jul 22 12:45:39.938045 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:39.937994 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" event={"ID":"8426be7e-d9b0-443f-8ed6-3e36d4f258ce","Type":"ContainerStarted","Data":"78b6c3ff4647b500db17fc4446da7a40d310d42b8cd7bc106d09325d1987a233"} Jul 22 12:45:39.961319 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:39.961259 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-qw8xc" podStartSLOduration=2.032991986 podStartE2EDuration="3.961242645s" totalImagesPullingTime="1.928250659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:45:39.959872386 +0000 UTC m=+293.764589296" watchObservedRunningTime="2026-07-22 12:45:39.961242645 +0000 UTC m=+293.765959540" Jul 22 12:45:40.894394 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.894363 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr"] Jul 22 12:45:40.897128 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.897111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:40.900341 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.900317 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 12:45:40.900438 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.900324 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 12:45:40.900988 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.900973 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f8jfx\"" Jul 22 12:45:40.908098 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.908072 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr"] Jul 22 12:45:40.966144 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.966125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:40.966462 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.966165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhhrh\" (UniqueName: \"kubernetes.io/projected/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-kube-api-access-nhhrh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:40.966462 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:40.966201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:41.066927 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.066907 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:41.067061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.066946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhhrh\" (UniqueName: \"kubernetes.io/projected/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-kube-api-access-nhhrh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:41.067061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.066984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:41.067286 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.067271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:41.067347 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.067332 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:41.076157 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.076129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhhrh\" (UniqueName: \"kubernetes.io/projected/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-kube-api-access-nhhrh\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:41.206131 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.206104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:41.338210 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.338187 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr"] Jul 22 12:45:41.339828 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:45:41.339806 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5e0cb1_0be7_433c_82d0_97fc4c5f5ba1.slice/crio-12011dda8bf60324f0d974705b5cb7aa5221e7164a8b8f7b23e7f02f0ff0ffca WatchSource:0}: Error finding container 12011dda8bf60324f0d974705b5cb7aa5221e7164a8b8f7b23e7f02f0ff0ffca: Status 404 returned error can't find the container with id 12011dda8bf60324f0d974705b5cb7aa5221e7164a8b8f7b23e7f02f0ff0ffca Jul 22 12:45:41.947967 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.947930 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerID="a314ba4fab9a7de8ff852fed850a674f158eea154357f9768cf8e5391dac1ec6" exitCode=0 Jul 22 12:45:41.948139 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.948000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" event={"ID":"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1","Type":"ContainerDied","Data":"a314ba4fab9a7de8ff852fed850a674f158eea154357f9768cf8e5391dac1ec6"} Jul 22 12:45:41.948139 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:41.948062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" event={"ID":"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1","Type":"ContainerStarted","Data":"12011dda8bf60324f0d974705b5cb7aa5221e7164a8b8f7b23e7f02f0ff0ffca"} Jul 22 12:45:43.057561 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.057524 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j"] Jul 22 12:45:43.059765 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.059743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" Jul 22 12:45:43.062366 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.062344 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 22 12:45:43.063248 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.063230 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 22 12:45:43.063248 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.063244 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-78rl4\"" Jul 22 12:45:43.071262 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.071241 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j"] Jul 22 12:45:43.184441 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.184407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tpm7j\" (UID: \"dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" Jul 22 12:45:43.184604 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.184483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rpql\" (UniqueName: \"kubernetes.io/projected/dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754-kube-api-access-9rpql\") pod \"cert-manager-cainjector-559cdd8c87-tpm7j\" (UID: \"dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" Jul 22 12:45:43.285027 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.284981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpql\" (UniqueName: \"kubernetes.io/projected/dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754-kube-api-access-9rpql\") pod \"cert-manager-cainjector-559cdd8c87-tpm7j\" (UID: \"dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" Jul 22 12:45:43.285203 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.285096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tpm7j\" (UID: \"dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" Jul 22 12:45:43.293932 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.293910 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-tpm7j\" (UID: \"dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" Jul 22 12:45:43.294122 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.294093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rpql\" (UniqueName: \"kubernetes.io/projected/dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754-kube-api-access-9rpql\") pod \"cert-manager-cainjector-559cdd8c87-tpm7j\" (UID: \"dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" Jul 22 12:45:43.376673 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.376609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" Jul 22 12:45:43.905317 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.905269 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j"] Jul 22 12:45:43.908605 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:45:43.908568 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd68f0b2_7cc9_48c9_8cb1_16ad13c8c754.slice/crio-27244e199bbed53ab84a48eae73f42568dd48418a6f90009be58a845da3f8d18 WatchSource:0}: Error finding container 27244e199bbed53ab84a48eae73f42568dd48418a6f90009be58a845da3f8d18: Status 404 returned error can't find the container with id 27244e199bbed53ab84a48eae73f42568dd48418a6f90009be58a845da3f8d18 Jul 22 12:45:43.958303 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.958278 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerID="52c58002fba8a4bff2a761d62b835ee9981198c600dbe2ae71b91b30e42f48f9" exitCode=0 Jul 22 12:45:43.958396 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.958358 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" event={"ID":"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1","Type":"ContainerDied","Data":"52c58002fba8a4bff2a761d62b835ee9981198c600dbe2ae71b91b30e42f48f9"} Jul 22 12:45:43.959539 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:43.959516 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" event={"ID":"dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754","Type":"ContainerStarted","Data":"27244e199bbed53ab84a48eae73f42568dd48418a6f90009be58a845da3f8d18"} Jul 22 12:45:44.966776 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:44.966745 2576 generic.go:356] "Generic (PLEG): container finished" podID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerID="a37b1017838588b8948a4f585e0a0eef24aa30ecaa68328647549aee86aebc26" exitCode=0 Jul 22 12:45:44.967209 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:44.966827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" event={"ID":"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1","Type":"ContainerDied","Data":"a37b1017838588b8948a4f585e0a0eef24aa30ecaa68328647549aee86aebc26"} Jul 22 12:45:46.409486 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.409464 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:46.514167 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.514140 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhhrh\" (UniqueName: \"kubernetes.io/projected/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-kube-api-access-nhhrh\") pod \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " Jul 22 12:45:46.514275 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.514182 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-util\") pod \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " Jul 22 12:45:46.514275 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.514209 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-bundle\") pod \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\" (UID: \"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1\") " Jul 22 12:45:46.514603 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.514576 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-bundle" (OuterVolumeSpecName: "bundle") pod "7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" (UID: "7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:45:46.516137 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.516112 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-kube-api-access-nhhrh" (OuterVolumeSpecName: "kube-api-access-nhhrh") pod "7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" (UID: "7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1"). InnerVolumeSpecName "kube-api-access-nhhrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:45:46.561612 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.561577 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-util" (OuterVolumeSpecName: "util") pod "7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" (UID: "7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:45:46.570279 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.570258 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:45:46.570652 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.570633 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:45:46.578914 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.578893 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:45:46.579321 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.579299 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:45:46.583543 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.583525 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 12:45:46.615267 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.615246 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhhrh\" (UniqueName: \"kubernetes.io/projected/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-kube-api-access-nhhrh\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:45:46.615267 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.615267 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-util\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:45:46.615385 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.615279 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1-bundle\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:45:46.977339 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.977259 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" Jul 22 12:45:46.977339 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.977269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frrwzr" event={"ID":"7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1","Type":"ContainerDied","Data":"12011dda8bf60324f0d974705b5cb7aa5221e7164a8b8f7b23e7f02f0ff0ffca"} Jul 22 12:45:46.977339 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.977327 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12011dda8bf60324f0d974705b5cb7aa5221e7164a8b8f7b23e7f02f0ff0ffca" Jul 22 12:45:46.978792 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:46.978766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" event={"ID":"dd68f0b2-7cc9-48c9-8cb1-16ad13c8c754","Type":"ContainerStarted","Data":"414e54c7360eb4e7a3746c69c5c71f8875c10545b6230890cfadb25809fb923e"} Jul 22 12:45:47.027062 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.027005 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-tpm7j" podStartSLOduration=1.508099404 podStartE2EDuration="4.026992039s" totalImagesPullingTime="2.518892635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:45:47.023854483 +0000 UTC m=+300.828571378" watchObservedRunningTime="2026-07-22 12:45:47.026992039 +0000 UTC m=+300.831708934" Jul 22 12:45:47.711165 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.711133 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-plj6l"] Jul 22 12:45:47.711504 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.711389 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerName="extract" Jul 22 12:45:47.711504 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.711400 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerName="extract" Jul 22 12:45:47.711504 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.711411 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerName="util" Jul 22 12:45:47.711504 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.711416 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerName="util" Jul 22 12:45:47.711504 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.711426 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerName="pull" Jul 22 12:45:47.711504 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.711431 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerName="pull" Jul 22 12:45:47.711504 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.711473 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5e0cb1-0be7-433c-82d0-97fc4c5f5ba1" containerName="extract" Jul 22 12:45:47.713065 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.713050 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:47.715512 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.715493 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5q6sn\"" Jul 22 12:45:47.729922 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.729897 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-plj6l"] Jul 22 12:45:47.826023 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.825996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2zcv\" (UniqueName: \"kubernetes.io/projected/ec18a240-09eb-4eb8-9abb-34d52637f319-kube-api-access-f2zcv\") pod \"cert-manager-webhook-6486ffd997-plj6l\" (UID: \"ec18a240-09eb-4eb8-9abb-34d52637f319\") " pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:47.826136 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.826112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec18a240-09eb-4eb8-9abb-34d52637f319-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-plj6l\" (UID: \"ec18a240-09eb-4eb8-9abb-34d52637f319\") " pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:47.926622 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.926595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec18a240-09eb-4eb8-9abb-34d52637f319-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-plj6l\" (UID: \"ec18a240-09eb-4eb8-9abb-34d52637f319\") " pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:47.926744 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.926653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2zcv\" (UniqueName: \"kubernetes.io/projected/ec18a240-09eb-4eb8-9abb-34d52637f319-kube-api-access-f2zcv\") pod \"cert-manager-webhook-6486ffd997-plj6l\" (UID: \"ec18a240-09eb-4eb8-9abb-34d52637f319\") " pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:47.937702 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.937674 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ec18a240-09eb-4eb8-9abb-34d52637f319-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-plj6l\" (UID: \"ec18a240-09eb-4eb8-9abb-34d52637f319\") " pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:47.937879 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:47.937857 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2zcv\" (UniqueName: \"kubernetes.io/projected/ec18a240-09eb-4eb8-9abb-34d52637f319-kube-api-access-f2zcv\") pod \"cert-manager-webhook-6486ffd997-plj6l\" (UID: \"ec18a240-09eb-4eb8-9abb-34d52637f319\") " pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:48.020931 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:48.020870 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:48.145244 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:48.145220 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-plj6l"] Jul 22 12:45:48.146767 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:45:48.146740 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec18a240_09eb_4eb8_9abb_34d52637f319.slice/crio-364e99693f7f72a9910dc0e958ce71ea31fc625176e28d783fbad3a8af07c76c WatchSource:0}: Error finding container 364e99693f7f72a9910dc0e958ce71ea31fc625176e28d783fbad3a8af07c76c: Status 404 returned error can't find the container with id 364e99693f7f72a9910dc0e958ce71ea31fc625176e28d783fbad3a8af07c76c Jul 22 12:45:48.988451 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:48.988412 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" event={"ID":"ec18a240-09eb-4eb8-9abb-34d52637f319","Type":"ContainerStarted","Data":"3180e3a2f88ea8aec13a7b2f59320b40126bb945d112656eb48ac2fc9d0a8607"} Jul 22 12:45:48.988451 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:48.988455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" event={"ID":"ec18a240-09eb-4eb8-9abb-34d52637f319","Type":"ContainerStarted","Data":"364e99693f7f72a9910dc0e958ce71ea31fc625176e28d783fbad3a8af07c76c"} Jul 22 12:45:48.988917 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:48.988479 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:45:49.011449 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:49.011408 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" podStartSLOduration=2.011394225 podStartE2EDuration="2.011394225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:45:49.010845169 +0000 UTC m=+302.815562064" watchObservedRunningTime="2026-07-22 12:45:49.011394225 +0000 UTC m=+302.816111119" Jul 22 12:45:54.995240 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:45:54.995214 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-plj6l" Jul 22 12:46:00.234849 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.234814 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vcwvw"] Jul 22 12:46:00.236967 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.236951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vcwvw" Jul 22 12:46:00.239259 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.239240 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2lx8m\"" Jul 22 12:46:00.248782 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.248755 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vcwvw"] Jul 22 12:46:00.316119 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.316097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51f7a22f-5bcf-4fb7-94a5-899d53c93278-bound-sa-token\") pod \"cert-manager-55b4d954d5-vcwvw\" (UID: \"51f7a22f-5bcf-4fb7-94a5-899d53c93278\") " pod="cert-manager/cert-manager-55b4d954d5-vcwvw" Jul 22 12:46:00.316226 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.316127 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld7t9\" (UniqueName: \"kubernetes.io/projected/51f7a22f-5bcf-4fb7-94a5-899d53c93278-kube-api-access-ld7t9\") pod \"cert-manager-55b4d954d5-vcwvw\" (UID: \"51f7a22f-5bcf-4fb7-94a5-899d53c93278\") " pod="cert-manager/cert-manager-55b4d954d5-vcwvw" Jul 22 12:46:00.417429 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.417400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51f7a22f-5bcf-4fb7-94a5-899d53c93278-bound-sa-token\") pod \"cert-manager-55b4d954d5-vcwvw\" (UID: \"51f7a22f-5bcf-4fb7-94a5-899d53c93278\") " pod="cert-manager/cert-manager-55b4d954d5-vcwvw" Jul 22 12:46:00.417535 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.417433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld7t9\" (UniqueName: \"kubernetes.io/projected/51f7a22f-5bcf-4fb7-94a5-899d53c93278-kube-api-access-ld7t9\") pod \"cert-manager-55b4d954d5-vcwvw\" (UID: \"51f7a22f-5bcf-4fb7-94a5-899d53c93278\") " pod="cert-manager/cert-manager-55b4d954d5-vcwvw" Jul 22 12:46:00.426885 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.426864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51f7a22f-5bcf-4fb7-94a5-899d53c93278-bound-sa-token\") pod \"cert-manager-55b4d954d5-vcwvw\" (UID: \"51f7a22f-5bcf-4fb7-94a5-899d53c93278\") " pod="cert-manager/cert-manager-55b4d954d5-vcwvw" Jul 22 12:46:00.426999 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.426957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld7t9\" (UniqueName: \"kubernetes.io/projected/51f7a22f-5bcf-4fb7-94a5-899d53c93278-kube-api-access-ld7t9\") pod \"cert-manager-55b4d954d5-vcwvw\" (UID: \"51f7a22f-5bcf-4fb7-94a5-899d53c93278\") " pod="cert-manager/cert-manager-55b4d954d5-vcwvw" Jul 22 12:46:00.545442 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.545384 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vcwvw" Jul 22 12:46:00.574095 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.574073 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt"] Jul 22 12:46:00.578702 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.578679 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.581179 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.581151 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 22 12:46:00.581295 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.581158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-f8jfx\"" Jul 22 12:46:00.581295 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.581226 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 22 12:46:00.587430 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.587399 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt"] Jul 22 12:46:00.674967 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.674944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vcwvw"] Jul 22 12:46:00.677286 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:46:00.677251 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51f7a22f_5bcf_4fb7_94a5_899d53c93278.slice/crio-e265f81efc245625079d948a7c83d6cb8a096200634021688be809af44d0dbed WatchSource:0}: Error finding container e265f81efc245625079d948a7c83d6cb8a096200634021688be809af44d0dbed: Status 404 returned error can't find the container with id e265f81efc245625079d948a7c83d6cb8a096200634021688be809af44d0dbed Jul 22 12:46:00.719761 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.719736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ccr5\" (UniqueName: \"kubernetes.io/projected/5d22c377-d638-4fe5-9c98-fdb25e403554-kube-api-access-2ccr5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.719848 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.719779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.719848 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.719824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.821241 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.821159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ccr5\" (UniqueName: \"kubernetes.io/projected/5d22c377-d638-4fe5-9c98-fdb25e403554-kube-api-access-2ccr5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.821561 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.821527 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.821760 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.821735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.821966 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.821944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.822422 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.822399 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.831490 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.831460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ccr5\" (UniqueName: \"kubernetes.io/projected/5d22c377-d638-4fe5-9c98-fdb25e403554-kube-api-access-2ccr5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:00.889383 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:00.889358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:01.039834 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:01.035351 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt"] Jul 22 12:46:01.042909 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:46:01.042882 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d22c377_d638_4fe5_9c98_fdb25e403554.slice/crio-03150da521a7efc1c295da80f4a7272d2b610ff023fd65011e5e07a0a7699347 WatchSource:0}: Error finding container 03150da521a7efc1c295da80f4a7272d2b610ff023fd65011e5e07a0a7699347: Status 404 returned error can't find the container with id 03150da521a7efc1c295da80f4a7272d2b610ff023fd65011e5e07a0a7699347 Jul 22 12:46:01.043300 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:01.043276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vcwvw" event={"ID":"51f7a22f-5bcf-4fb7-94a5-899d53c93278","Type":"ContainerStarted","Data":"3a8248702d329ed5b741e4d1c6a3f1f044d124ea4e8a188df69f7ad6e7f730a6"} Jul 22 12:46:01.043388 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:01.043315 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vcwvw" event={"ID":"51f7a22f-5bcf-4fb7-94a5-899d53c93278","Type":"ContainerStarted","Data":"e265f81efc245625079d948a7c83d6cb8a096200634021688be809af44d0dbed"} Jul 22 12:46:01.061840 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:01.061798 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vcwvw" podStartSLOduration=1.061785201 podStartE2EDuration="1.061785201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:01.060692368 +0000 UTC m=+314.865409265" watchObservedRunningTime="2026-07-22 12:46:01.061785201 +0000 UTC m=+314.866502097" Jul 22 12:46:02.048870 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:02.048840 2576 generic.go:356] "Generic (PLEG): container finished" podID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerID="15c8750133229d5ed9e372cd8cfc493e3a77ef9fe43b928d01fc6190eb147478" exitCode=0 Jul 22 12:46:02.049375 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:02.048931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" event={"ID":"5d22c377-d638-4fe5-9c98-fdb25e403554","Type":"ContainerDied","Data":"15c8750133229d5ed9e372cd8cfc493e3a77ef9fe43b928d01fc6190eb147478"} Jul 22 12:46:02.049375 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:02.048993 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" event={"ID":"5d22c377-d638-4fe5-9c98-fdb25e403554","Type":"ContainerStarted","Data":"03150da521a7efc1c295da80f4a7272d2b610ff023fd65011e5e07a0a7699347"} Jul 22 12:46:02.049375 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:02.049343 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:46:03.054689 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:03.054651 2576 generic.go:356] "Generic (PLEG): container finished" podID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerID="32d30627af45b965f999282a9a4b3979f48948c529467178d989922f638c8a3e" exitCode=0 Jul 22 12:46:03.055084 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:03.054695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" event={"ID":"5d22c377-d638-4fe5-9c98-fdb25e403554","Type":"ContainerDied","Data":"32d30627af45b965f999282a9a4b3979f48948c529467178d989922f638c8a3e"} Jul 22 12:46:04.061062 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:04.061030 2576 generic.go:356] "Generic (PLEG): container finished" podID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerID="be2e2dadc1f2886440a9da65385afe281189711e9e3b1f65b2158f9115ef32e1" exitCode=0 Jul 22 12:46:04.061416 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:04.061108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" event={"ID":"5d22c377-d638-4fe5-9c98-fdb25e403554","Type":"ContainerDied","Data":"be2e2dadc1f2886440a9da65385afe281189711e9e3b1f65b2158f9115ef32e1"} Jul 22 12:46:05.180461 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.180434 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:05.254712 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.254690 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ccr5\" (UniqueName: \"kubernetes.io/projected/5d22c377-d638-4fe5-9c98-fdb25e403554-kube-api-access-2ccr5\") pod \"5d22c377-d638-4fe5-9c98-fdb25e403554\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " Jul 22 12:46:05.254841 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.254722 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-bundle\") pod \"5d22c377-d638-4fe5-9c98-fdb25e403554\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " Jul 22 12:46:05.254841 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.254777 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-util\") pod \"5d22c377-d638-4fe5-9c98-fdb25e403554\" (UID: \"5d22c377-d638-4fe5-9c98-fdb25e403554\") " Jul 22 12:46:05.255420 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.255397 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-bundle" (OuterVolumeSpecName: "bundle") pod "5d22c377-d638-4fe5-9c98-fdb25e403554" (UID: "5d22c377-d638-4fe5-9c98-fdb25e403554"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:46:05.256832 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.256805 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d22c377-d638-4fe5-9c98-fdb25e403554-kube-api-access-2ccr5" (OuterVolumeSpecName: "kube-api-access-2ccr5") pod "5d22c377-d638-4fe5-9c98-fdb25e403554" (UID: "5d22c377-d638-4fe5-9c98-fdb25e403554"). InnerVolumeSpecName "kube-api-access-2ccr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:46:05.260495 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.260412 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-util" (OuterVolumeSpecName: "util") pod "5d22c377-d638-4fe5-9c98-fdb25e403554" (UID: "5d22c377-d638-4fe5-9c98-fdb25e403554"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:46:05.355728 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.355680 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-bundle\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:46:05.355728 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.355699 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d22c377-d638-4fe5-9c98-fdb25e403554-util\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:46:05.355728 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:05.355708 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ccr5\" (UniqueName: \"kubernetes.io/projected/5d22c377-d638-4fe5-9c98-fdb25e403554-kube-api-access-2ccr5\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:46:06.071072 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:06.071042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" event={"ID":"5d22c377-d638-4fe5-9c98-fdb25e403554","Type":"ContainerDied","Data":"03150da521a7efc1c295da80f4a7272d2b610ff023fd65011e5e07a0a7699347"} Jul 22 12:46:06.071208 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:06.071089 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03150da521a7efc1c295da80f4a7272d2b610ff023fd65011e5e07a0a7699347" Jul 22 12:46:06.071208 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:06.071057 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af5zqmt" Jul 22 12:46:17.589437 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.589404 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs"] Jul 22 12:46:17.589886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.589687 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerName="util" Jul 22 12:46:17.589886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.589697 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerName="util" Jul 22 12:46:17.589886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.589718 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerName="pull" Jul 22 12:46:17.589886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.589723 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerName="pull" Jul 22 12:46:17.589886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.589733 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerName="extract" Jul 22 12:46:17.589886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.589738 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerName="extract" Jul 22 12:46:17.589886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.589788 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d22c377-d638-4fe5-9c98-fdb25e403554" containerName="extract" Jul 22 12:46:17.593925 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.593906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.597297 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.597272 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 12:46:17.597411 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.597249 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nqhtb\"" Jul 22 12:46:17.597622 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.597604 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 12:46:17.597675 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.597607 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 12:46:17.597733 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.597614 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 12:46:17.625744 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.625723 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs"] Jul 22 12:46:17.747846 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.747819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nql7p\" (UniqueName: \"kubernetes.io/projected/55374e50-ebda-4d7d-ad74-953f5ba81c9c-kube-api-access-nql7p\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.747985 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.747858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55374e50-ebda-4d7d-ad74-953f5ba81c9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.747985 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.747896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55374e50-ebda-4d7d-ad74-953f5ba81c9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.747985 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.747938 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55374e50-ebda-4d7d-ad74-953f5ba81c9c-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.747985 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.747980 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55374e50-ebda-4d7d-ad74-953f5ba81c9c-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.849475 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.849395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nql7p\" (UniqueName: \"kubernetes.io/projected/55374e50-ebda-4d7d-ad74-953f5ba81c9c-kube-api-access-nql7p\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.849475 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.849433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55374e50-ebda-4d7d-ad74-953f5ba81c9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.849475 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.849457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55374e50-ebda-4d7d-ad74-953f5ba81c9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.849711 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.849482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55374e50-ebda-4d7d-ad74-953f5ba81c9c-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.849711 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.849538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55374e50-ebda-4d7d-ad74-953f5ba81c9c-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.849955 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.849923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55374e50-ebda-4d7d-ad74-953f5ba81c9c-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.851932 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.851910 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55374e50-ebda-4d7d-ad74-953f5ba81c9c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.852060 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.851970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55374e50-ebda-4d7d-ad74-953f5ba81c9c-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.852179 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.852162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55374e50-ebda-4d7d-ad74-953f5ba81c9c-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.858733 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.858711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nql7p\" (UniqueName: \"kubernetes.io/projected/55374e50-ebda-4d7d-ad74-953f5ba81c9c-kube-api-access-nql7p\") pod \"opendatahub-operator-controller-manager-64bbff868f-8ztbs\" (UID: \"55374e50-ebda-4d7d-ad74-953f5ba81c9c\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:17.903689 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:17.903662 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:18.071865 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:18.071776 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs"] Jul 22 12:46:18.073732 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:46:18.073706 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55374e50_ebda_4d7d_ad74_953f5ba81c9c.slice/crio-06db3b94af2c28f1e66d6cd2f7cac60c36b10bf19e07e3f302ca94af64c01715 WatchSource:0}: Error finding container 06db3b94af2c28f1e66d6cd2f7cac60c36b10bf19e07e3f302ca94af64c01715: Status 404 returned error can't find the container with id 06db3b94af2c28f1e66d6cd2f7cac60c36b10bf19e07e3f302ca94af64c01715 Jul 22 12:46:18.122054 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:18.121966 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" event={"ID":"55374e50-ebda-4d7d-ad74-953f5ba81c9c","Type":"ContainerStarted","Data":"06db3b94af2c28f1e66d6cd2f7cac60c36b10bf19e07e3f302ca94af64c01715"} Jul 22 12:46:21.141759 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:21.141728 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" event={"ID":"55374e50-ebda-4d7d-ad74-953f5ba81c9c","Type":"ContainerStarted","Data":"97ff673d34f2f576fe0f4b7155d5d4732d7c390dc0bfc7cc992c09c0176e4da8"} Jul 22 12:46:22.147325 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:22.147290 2576 generic.go:356] "Generic (PLEG): container finished" podID="55374e50-ebda-4d7d-ad74-953f5ba81c9c" containerID="97ff673d34f2f576fe0f4b7155d5d4732d7c390dc0bfc7cc992c09c0176e4da8" exitCode=0 Jul 22 12:46:22.147719 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:22.147350 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" event={"ID":"55374e50-ebda-4d7d-ad74-953f5ba81c9c","Type":"ContainerDied","Data":"97ff673d34f2f576fe0f4b7155d5d4732d7c390dc0bfc7cc992c09c0176e4da8"} Jul 22 12:46:23.154149 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:23.154062 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" event={"ID":"55374e50-ebda-4d7d-ad74-953f5ba81c9c","Type":"ContainerStarted","Data":"fedab6f11b20731cdd50af746e1bf9d567a97cf2b5699ed45b28216b63f4fd4d"} Jul 22 12:46:23.154581 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:23.154179 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:23.180795 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:23.180748 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" podStartSLOduration=1.784266132 podStartE2EDuration="6.180729657s" totalImagesPullingTime="3.394010734s" totalInitContainerRuntime="1.002452791s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:23.179559675 +0000 UTC m=+336.984276570" watchObservedRunningTime="2026-07-22 12:46:23.180729657 +0000 UTC m=+336.985446552" Jul 22 12:46:34.161138 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:34.161111 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-8ztbs" Jul 22 12:46:37.246130 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.246047 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h"] Jul 22 12:46:37.249495 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.249474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.255338 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.255289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 22 12:46:37.256234 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.256213 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 22 12:46:37.256331 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.256213 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-v4ckr\"" Jul 22 12:46:37.271188 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.271158 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h"] Jul 22 12:46:37.308956 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.308920 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lkz\" (UniqueName: \"kubernetes.io/projected/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-kube-api-access-q9lkz\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.309157 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.308963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-tmp\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.309157 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.309129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-tls-certs\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.410562 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.410523 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-tls-certs\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.410730 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.410623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lkz\" (UniqueName: \"kubernetes.io/projected/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-kube-api-access-q9lkz\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.410730 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.410671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-tmp\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.413145 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.413119 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-tmp\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.413351 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.413335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-tls-certs\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.436407 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.436381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lkz\" (UniqueName: \"kubernetes.io/projected/17c4a082-5d5a-4e10-aec1-5e2f88fa12fb-kube-api-access-q9lkz\") pod \"kube-auth-proxy-76c9b95cb4-6pb8h\" (UID: \"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb\") " pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.558713 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.558642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" Jul 22 12:46:37.711326 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:37.711286 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h"] Jul 22 12:46:37.713211 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:46:37.713181 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c4a082_5d5a_4e10_aec1_5e2f88fa12fb.slice/crio-ad1fd56896f1ee5f54b7f16e0adc5c7ca9374a1ce00b902a993211944e1baceb WatchSource:0}: Error finding container ad1fd56896f1ee5f54b7f16e0adc5c7ca9374a1ce00b902a993211944e1baceb: Status 404 returned error can't find the container with id ad1fd56896f1ee5f54b7f16e0adc5c7ca9374a1ce00b902a993211944e1baceb Jul 22 12:46:38.219828 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.219794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" event={"ID":"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb","Type":"ContainerStarted","Data":"ad1fd56896f1ee5f54b7f16e0adc5c7ca9374a1ce00b902a993211944e1baceb"} Jul 22 12:46:38.757229 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.757195 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-svf46"] Jul 22 12:46:38.760773 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.760748 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:38.765666 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.765632 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dl4bh\"" Jul 22 12:46:38.777725 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.777697 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 22 12:46:38.820144 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.820106 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-svf46"] Jul 22 12:46:38.822416 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.822344 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f9501f-bf05-47e3-ae81-c99b2a69e25c-tls-certs\") pod \"model-serving-api-54d89bd5fc-svf46\" (UID: \"07f9501f-bf05-47e3-ae81-c99b2a69e25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:38.822416 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.822388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw8s7\" (UniqueName: \"kubernetes.io/projected/07f9501f-bf05-47e3-ae81-c99b2a69e25c-kube-api-access-vw8s7\") pod \"model-serving-api-54d89bd5fc-svf46\" (UID: \"07f9501f-bf05-47e3-ae81-c99b2a69e25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:38.923256 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.923224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f9501f-bf05-47e3-ae81-c99b2a69e25c-tls-certs\") pod \"model-serving-api-54d89bd5fc-svf46\" (UID: \"07f9501f-bf05-47e3-ae81-c99b2a69e25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:38.923256 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.923259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw8s7\" (UniqueName: \"kubernetes.io/projected/07f9501f-bf05-47e3-ae81-c99b2a69e25c-kube-api-access-vw8s7\") pod \"model-serving-api-54d89bd5fc-svf46\" (UID: \"07f9501f-bf05-47e3-ae81-c99b2a69e25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:38.926163 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.926129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07f9501f-bf05-47e3-ae81-c99b2a69e25c-tls-certs\") pod \"model-serving-api-54d89bd5fc-svf46\" (UID: \"07f9501f-bf05-47e3-ae81-c99b2a69e25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:38.945032 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:38.944979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw8s7\" (UniqueName: \"kubernetes.io/projected/07f9501f-bf05-47e3-ae81-c99b2a69e25c-kube-api-access-vw8s7\") pod \"model-serving-api-54d89bd5fc-svf46\" (UID: \"07f9501f-bf05-47e3-ae81-c99b2a69e25c\") " pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:39.073627 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:39.073536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:39.247048 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:39.247001 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-svf46"] Jul 22 12:46:39.250319 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:46:39.250286 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f9501f_bf05_47e3_ae81_c99b2a69e25c.slice/crio-0a253aacb1417ac2f271cffb64a412b83f1dc5a8091a76c5c8209b034aa115bf WatchSource:0}: Error finding container 0a253aacb1417ac2f271cffb64a412b83f1dc5a8091a76c5c8209b034aa115bf: Status 404 returned error can't find the container with id 0a253aacb1417ac2f271cffb64a412b83f1dc5a8091a76c5c8209b034aa115bf Jul 22 12:46:40.231223 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:40.231183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-svf46" event={"ID":"07f9501f-bf05-47e3-ae81-c99b2a69e25c","Type":"ContainerStarted","Data":"0a253aacb1417ac2f271cffb64a412b83f1dc5a8091a76c5c8209b034aa115bf"} Jul 22 12:46:40.233049 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:40.233002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" event={"ID":"17c4a082-5d5a-4e10-aec1-5e2f88fa12fb","Type":"ContainerStarted","Data":"7f6545e2c8284227536539450047b3f4e88ee82646a68b40014cc131ea8cbac7"} Jul 22 12:46:40.263512 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:40.263452 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76c9b95cb4-6pb8h" podStartSLOduration=0.87142863 podStartE2EDuration="3.263429628s" totalImagesPullingTime="2.392000998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:40.262936651 +0000 UTC m=+354.067653539" watchObservedRunningTime="2026-07-22 12:46:40.263429628 +0000 UTC m=+354.068146524" Jul 22 12:46:41.239686 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:41.239651 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-svf46" event={"ID":"07f9501f-bf05-47e3-ae81-c99b2a69e25c","Type":"ContainerStarted","Data":"1deec91efd57e6f405cda95099af7b5917b1b481eb6aff136b0f4884be364eaf"} Jul 22 12:46:41.264734 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:41.264688 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-svf46" podStartSLOduration=1.375044067 podStartE2EDuration="3.264671524s" totalImagesPullingTime="1.889627457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:41.263481557 +0000 UTC m=+355.068198463" watchObservedRunningTime="2026-07-22 12:46:41.264671524 +0000 UTC m=+355.069388420" Jul 22 12:46:42.245482 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:42.245448 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:48.688342 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.688303 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f"] Jul 22 12:46:48.693065 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.693049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.695867 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.695840 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 12:46:48.697838 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.697819 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 12:46:48.698143 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.698122 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 12:46:48.698240 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.698131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m6lkm\"" Jul 22 12:46:48.698374 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.698354 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 12:46:48.721128 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.721107 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f"] Jul 22 12:46:48.806940 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.806914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.807075 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.806955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87215607-2efc-4c09-82f1-d9ef64a78026-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.807075 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.806972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87215607-2efc-4c09-82f1-d9ef64a78026-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.807075 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.807001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mhn\" (UniqueName: \"kubernetes.io/projected/87215607-2efc-4c09-82f1-d9ef64a78026-kube-api-access-s9mhn\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.807075 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.807067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.807239 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.807100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.807239 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.807121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87215607-2efc-4c09-82f1-d9ef64a78026-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.908414 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.908385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87215607-2efc-4c09-82f1-d9ef64a78026-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.908617 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.908424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87215607-2efc-4c09-82f1-d9ef64a78026-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.908617 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.908579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mhn\" (UniqueName: \"kubernetes.io/projected/87215607-2efc-4c09-82f1-d9ef64a78026-kube-api-access-s9mhn\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.908740 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.908631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.908740 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.908677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.908740 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.908709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87215607-2efc-4c09-82f1-d9ef64a78026-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.908886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.908742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.911354 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.910078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87215607-2efc-4c09-82f1-d9ef64a78026-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.911918 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.911888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.917204 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.917087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87215607-2efc-4c09-82f1-d9ef64a78026-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.917204 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.917163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.917380 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.917262 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87215607-2efc-4c09-82f1-d9ef64a78026-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.928451 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.928429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87215607-2efc-4c09-82f1-d9ef64a78026-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:48.932152 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:48.932131 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mhn\" (UniqueName: \"kubernetes.io/projected/87215607-2efc-4c09-82f1-d9ef64a78026-kube-api-access-s9mhn\") pod \"istiod-openshift-gateway-94bb8fbfd-vqg5f\" (UID: \"87215607-2efc-4c09-82f1-d9ef64a78026\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:49.002235 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:49.002165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:49.175872 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:49.175842 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f"] Jul 22 12:46:49.177154 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:46:49.177129 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87215607_2efc_4c09_82f1_d9ef64a78026.slice/crio-a02231815dbcdeb08ba8bf1e0baaf3662ed0b684a3876baed0303abeefd12f5c WatchSource:0}: Error finding container a02231815dbcdeb08ba8bf1e0baaf3662ed0b684a3876baed0303abeefd12f5c: Status 404 returned error can't find the container with id a02231815dbcdeb08ba8bf1e0baaf3662ed0b684a3876baed0303abeefd12f5c Jul 22 12:46:49.282151 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:49.282072 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" event={"ID":"87215607-2efc-4c09-82f1-d9ef64a78026","Type":"ContainerStarted","Data":"a02231815dbcdeb08ba8bf1e0baaf3662ed0b684a3876baed0303abeefd12f5c"} Jul 22 12:46:50.532215 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.532183 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc"] Jul 22 12:46:50.548179 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.548154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:50.551603 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.551580 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pzjwz\"" Jul 22 12:46:50.555565 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.555543 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 22 12:46:50.574501 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.574446 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc"] Jul 22 12:46:50.594170 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.594142 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc"] Jul 22 12:46:50.594285 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.594250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:50.598091 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.598070 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 22 12:46:50.598193 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.598162 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nhz7p\"" Jul 22 12:46:50.609152 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.609131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc"] Jul 22 12:46:50.624322 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.624301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe-cert\") pod \"kserve-controller-manager-d6f4b65b8-vz6lc\" (UID: \"f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:50.624406 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.624336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qqdf\" (UniqueName: \"kubernetes.io/projected/f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe-kube-api-access-2qqdf\") pod \"kserve-controller-manager-d6f4b65b8-vz6lc\" (UID: \"f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:50.725260 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.725228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/380a3bc1-3f05-496d-834c-a7df7a8104ee-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66ssc\" (UID: \"380a3bc1-3f05-496d-834c-a7df7a8104ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:50.725422 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.725272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe-cert\") pod \"kserve-controller-manager-d6f4b65b8-vz6lc\" (UID: \"f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:50.725422 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.725307 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qqdf\" (UniqueName: \"kubernetes.io/projected/f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe-kube-api-access-2qqdf\") pod \"kserve-controller-manager-d6f4b65b8-vz6lc\" (UID: \"f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:50.725422 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.725396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44cv9\" (UniqueName: \"kubernetes.io/projected/380a3bc1-3f05-496d-834c-a7df7a8104ee-kube-api-access-44cv9\") pod \"llmisvc-controller-manager-777c6bd7d5-66ssc\" (UID: \"380a3bc1-3f05-496d-834c-a7df7a8104ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:50.727831 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.727776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe-cert\") pod \"kserve-controller-manager-d6f4b65b8-vz6lc\" (UID: \"f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:50.739889 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.739864 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qqdf\" (UniqueName: \"kubernetes.io/projected/f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe-kube-api-access-2qqdf\") pod \"kserve-controller-manager-d6f4b65b8-vz6lc\" (UID: \"f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:50.826062 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.825974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44cv9\" (UniqueName: \"kubernetes.io/projected/380a3bc1-3f05-496d-834c-a7df7a8104ee-kube-api-access-44cv9\") pod \"llmisvc-controller-manager-777c6bd7d5-66ssc\" (UID: \"380a3bc1-3f05-496d-834c-a7df7a8104ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:50.826062 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.826045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/380a3bc1-3f05-496d-834c-a7df7a8104ee-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66ssc\" (UID: \"380a3bc1-3f05-496d-834c-a7df7a8104ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:50.826263 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:46:50.826193 2576 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 22 12:46:50.826263 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:46:50.826254 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380a3bc1-3f05-496d-834c-a7df7a8104ee-cert podName:380a3bc1-3f05-496d-834c-a7df7a8104ee nodeName:}" failed. No retries permitted until 2026-07-22 12:46:51.326234085 +0000 UTC m=+365.130950970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/380a3bc1-3f05-496d-834c-a7df7a8104ee-cert") pod "llmisvc-controller-manager-777c6bd7d5-66ssc" (UID: "380a3bc1-3f05-496d-834c-a7df7a8104ee") : secret "llmisvc-webhook-server-cert" not found Jul 22 12:46:50.838920 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.838889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44cv9\" (UniqueName: \"kubernetes.io/projected/380a3bc1-3f05-496d-834c-a7df7a8104ee-kube-api-access-44cv9\") pod \"llmisvc-controller-manager-777c6bd7d5-66ssc\" (UID: \"380a3bc1-3f05-496d-834c-a7df7a8104ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:50.860558 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:50.860537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:51.052207 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:51.052173 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc"] Jul 22 12:46:51.331236 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:51.331203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/380a3bc1-3f05-496d-834c-a7df7a8104ee-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66ssc\" (UID: \"380a3bc1-3f05-496d-834c-a7df7a8104ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:51.334651 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:51.334627 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/380a3bc1-3f05-496d-834c-a7df7a8104ee-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-66ssc\" (UID: \"380a3bc1-3f05-496d-834c-a7df7a8104ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:51.503684 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:51.503650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:51.776444 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:46:51.776400 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59f1c5c_bf40_4b22_9cf6_fd36bb0e0abe.slice/crio-4e2357f93fc3f6f26e8db899108cd783072dda41fcecad63c577dcae296fd687 WatchSource:0}: Error finding container 4e2357f93fc3f6f26e8db899108cd783072dda41fcecad63c577dcae296fd687: Status 404 returned error can't find the container with id 4e2357f93fc3f6f26e8db899108cd783072dda41fcecad63c577dcae296fd687 Jul 22 12:46:51.839362 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:51.839325 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 12:46:51.839462 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:51.839398 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 12:46:51.921174 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:51.921151 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc"] Jul 22 12:46:51.922892 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:46:51.922860 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod380a3bc1_3f05_496d_834c_a7df7a8104ee.slice/crio-94c73f162bca4833ed7b95940f3aa5bb48909f039cfb254351d35d81cfdcee13 WatchSource:0}: Error finding container 94c73f162bca4833ed7b95940f3aa5bb48909f039cfb254351d35d81cfdcee13: Status 404 returned error can't find the container with id 94c73f162bca4833ed7b95940f3aa5bb48909f039cfb254351d35d81cfdcee13 Jul 22 12:46:52.301079 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:52.301040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" event={"ID":"f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe","Type":"ContainerStarted","Data":"4e2357f93fc3f6f26e8db899108cd783072dda41fcecad63c577dcae296fd687"} Jul 22 12:46:52.302348 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:52.302316 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" event={"ID":"380a3bc1-3f05-496d-834c-a7df7a8104ee","Type":"ContainerStarted","Data":"94c73f162bca4833ed7b95940f3aa5bb48909f039cfb254351d35d81cfdcee13"} Jul 22 12:46:52.303887 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:52.303854 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" event={"ID":"87215607-2efc-4c09-82f1-d9ef64a78026","Type":"ContainerStarted","Data":"239429a8c3dea9870ef209c8759b99e42e5bca9bd2abead0ec1e44b590024fe3"} Jul 22 12:46:52.304069 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:52.304053 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:46:52.305894 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:52.305870 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-vqg5f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 12:46:52.305993 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:52.305924 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" podUID="87215607-2efc-4c09-82f1-d9ef64a78026" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 12:46:52.347789 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:52.347741 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" podStartSLOduration=1.6877796649999999 podStartE2EDuration="4.34772522s" totalImagesPullingTime="2.659945555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:52.345646175 +0000 UTC m=+366.150363072" watchObservedRunningTime="2026-07-22 12:46:52.34772522 +0000 UTC m=+366.152442115" Jul 22 12:46:53.255228 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:53.255193 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-svf46" Jul 22 12:46:53.311457 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:53.311416 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-vqg5f container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 12:46:53.311626 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:53.311475 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" podUID="87215607-2efc-4c09-82f1-d9ef64a78026" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 12:46:55.321447 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:55.321403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" event={"ID":"f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe","Type":"ContainerStarted","Data":"52d9b143d2ca840574c27fbc1246953674ed3eb212a5d865b606ebbad8978d5c"} Jul 22 12:46:55.321933 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:55.321485 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:46:55.322907 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:55.322881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" event={"ID":"380a3bc1-3f05-496d-834c-a7df7a8104ee","Type":"ContainerStarted","Data":"e6dc06f016de6ad8923d165cd863d624ca2faedad8365952a249105ee78369fe"} Jul 22 12:46:55.323041 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:55.323024 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:46:55.345630 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:55.345577 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" podStartSLOduration=2.34117776 podStartE2EDuration="5.345562495s" totalImagesPullingTime="3.004384735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:55.344797005 +0000 UTC m=+369.149513904" watchObservedRunningTime="2026-07-22 12:46:55.345562495 +0000 UTC m=+369.150279391" Jul 22 12:46:55.365482 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:55.365432 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" podStartSLOduration=2.461738737 podStartE2EDuration="5.365417677s" totalImagesPullingTime="2.90367894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:55.364122108 +0000 UTC m=+369.168839017" watchObservedRunningTime="2026-07-22 12:46:55.365417677 +0000 UTC m=+369.170134573" Jul 22 12:46:56.312109 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:46:56.312077 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-vqg5f" Jul 22 12:47:26.329444 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:26.329414 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-66ssc" Jul 22 12:47:26.332391 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:26.332372 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vz6lc" Jul 22 12:47:54.246407 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.246369 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf"] Jul 22 12:47:54.249854 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.249836 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.253432 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.253408 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g556c\"" Jul 22 12:47:54.254511 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.254492 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 12:47:54.254623 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.254557 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 12:47:54.273766 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.273734 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf"] Jul 22 12:47:54.348611 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.348575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.348611 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.348615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.348869 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.348758 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrsq\" (UniqueName: \"kubernetes.io/projected/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-kube-api-access-zxrsq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.449587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.449548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.449587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.449587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.449793 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.449653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrsq\" (UniqueName: \"kubernetes.io/projected/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-kube-api-access-zxrsq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.449942 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.449922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.450006 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.449988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.462068 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.462039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrsq\" (UniqueName: \"kubernetes.io/projected/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-kube-api-access-zxrsq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.559220 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.559153 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:54.620238 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.620201 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv"] Jul 22 12:47:54.624379 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.624359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.638837 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.638814 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv"] Jul 22 12:47:54.700758 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.700734 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf"] Jul 22 12:47:54.702811 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:47:54.702782 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2bfb32_d761_4047_8be9_86e52e4ec9fb.slice/crio-ddcf86700f0604046ba267fb1152a4c1f54cd8e794393d7f71bd04f27d329877 WatchSource:0}: Error finding container ddcf86700f0604046ba267fb1152a4c1f54cd8e794393d7f71bd04f27d329877: Status 404 returned error can't find the container with id ddcf86700f0604046ba267fb1152a4c1f54cd8e794393d7f71bd04f27d329877 Jul 22 12:47:54.751066 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.751041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.751151 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.751103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.751151 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.751141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdmbc\" (UniqueName: \"kubernetes.io/projected/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-kube-api-access-mdmbc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.852362 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.852293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.852362 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.852346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.852529 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.852372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdmbc\" (UniqueName: \"kubernetes.io/projected/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-kube-api-access-mdmbc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.852659 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.852638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.852691 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.852673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.864178 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.864155 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdmbc\" (UniqueName: \"kubernetes.io/projected/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-kube-api-access-mdmbc\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:54.933739 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:54.933708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:55.023903 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.023871 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn"] Jul 22 12:47:55.028323 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.028295 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.045337 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.045301 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn"] Jul 22 12:47:55.076612 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.076588 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv"] Jul 22 12:47:55.078478 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:47:55.078455 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f31666c_b4a9_4e7e_b2df_29bf5be874aa.slice/crio-4c908012bd05a8cb59dfa279741022370d712b3337ab6702fd7ca2aacb76d593 WatchSource:0}: Error finding container 4c908012bd05a8cb59dfa279741022370d712b3337ab6702fd7ca2aacb76d593: Status 404 returned error can't find the container with id 4c908012bd05a8cb59dfa279741022370d712b3337ab6702fd7ca2aacb76d593 Jul 22 12:47:55.154365 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.154323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.154365 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.154372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gf6\" (UniqueName: \"kubernetes.io/projected/ea53e9ad-7714-494f-b698-e164671c6ab7-kube-api-access-l9gf6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.154568 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.154391 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.255573 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.255533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.256059 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.255586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gf6\" (UniqueName: \"kubernetes.io/projected/ea53e9ad-7714-494f-b698-e164671c6ab7-kube-api-access-l9gf6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.256059 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.255612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.256059 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.255897 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.256059 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.255959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.265612 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.265591 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gf6\" (UniqueName: \"kubernetes.io/projected/ea53e9ad-7714-494f-b698-e164671c6ab7-kube-api-access-l9gf6\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.352603 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.352580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:55.479183 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.479158 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn"] Jul 22 12:47:55.481080 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:47:55.481055 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea53e9ad_7714_494f_b698_e164671c6ab7.slice/crio-026c4c7544ecf58c6cc96624e50d6f1ad044e73b248c140d06fbdd5186530c8d WatchSource:0}: Error finding container 026c4c7544ecf58c6cc96624e50d6f1ad044e73b248c140d06fbdd5186530c8d: Status 404 returned error can't find the container with id 026c4c7544ecf58c6cc96624e50d6f1ad044e73b248c140d06fbdd5186530c8d Jul 22 12:47:55.599663 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.599637 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerID="fe031a5f8e9221b8dc1705947e6a7caf1e80ed58ffe43a90d3565763e6937ad1" exitCode=0 Jul 22 12:47:55.599811 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.599719 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" event={"ID":"0d2bfb32-d761-4047-8be9-86e52e4ec9fb","Type":"ContainerDied","Data":"fe031a5f8e9221b8dc1705947e6a7caf1e80ed58ffe43a90d3565763e6937ad1"} Jul 22 12:47:55.599811 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.599764 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" event={"ID":"0d2bfb32-d761-4047-8be9-86e52e4ec9fb","Type":"ContainerStarted","Data":"ddcf86700f0604046ba267fb1152a4c1f54cd8e794393d7f71bd04f27d329877"} Jul 22 12:47:55.601367 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.601349 2576 generic.go:356] "Generic (PLEG): container finished" podID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerID="f17b87ef071f05babc7ef9cdb80bd5e0758efaa54657cafa63721a4dd158f865" exitCode=0 Jul 22 12:47:55.601460 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.601428 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" event={"ID":"ea53e9ad-7714-494f-b698-e164671c6ab7","Type":"ContainerDied","Data":"f17b87ef071f05babc7ef9cdb80bd5e0758efaa54657cafa63721a4dd158f865"} Jul 22 12:47:55.601514 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.601468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" event={"ID":"ea53e9ad-7714-494f-b698-e164671c6ab7","Type":"ContainerStarted","Data":"026c4c7544ecf58c6cc96624e50d6f1ad044e73b248c140d06fbdd5186530c8d"} Jul 22 12:47:55.602850 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.602829 2576 generic.go:356] "Generic (PLEG): container finished" podID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerID="19ddb8faa9188175248a277b34e9ddc8d49aafcac6367c61e723738d2bb973c0" exitCode=0 Jul 22 12:47:55.602938 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.602859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" event={"ID":"3f31666c-b4a9-4e7e-b2df-29bf5be874aa","Type":"ContainerDied","Data":"19ddb8faa9188175248a277b34e9ddc8d49aafcac6367c61e723738d2bb973c0"} Jul 22 12:47:55.602938 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.602883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" event={"ID":"3f31666c-b4a9-4e7e-b2df-29bf5be874aa","Type":"ContainerStarted","Data":"4c908012bd05a8cb59dfa279741022370d712b3337ab6702fd7ca2aacb76d593"} Jul 22 12:47:55.621591 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.621537 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c"] Jul 22 12:47:55.624893 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.624878 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.634545 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.634522 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c"] Jul 22 12:47:55.660640 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.660618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkddp\" (UniqueName: \"kubernetes.io/projected/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-kube-api-access-pkddp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.660801 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.660779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.660843 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.660827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.761360 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.761326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkddp\" (UniqueName: \"kubernetes.io/projected/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-kube-api-access-pkddp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.761568 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.761509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.761568 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.761564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.761880 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.761861 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.761924 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.761883 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.774157 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.774128 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkddp\" (UniqueName: \"kubernetes.io/projected/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-kube-api-access-pkddp\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:55.954449 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:55.954416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:56.110572 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:56.110548 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c"] Jul 22 12:47:56.111596 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:47:56.111573 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7d0a0a_964f_4ede_ab83_83abdb9a9525.slice/crio-1618af8b7e1d720b29faf81592f268ba13dc798cb7138ddce5c95c8a65a823f9 WatchSource:0}: Error finding container 1618af8b7e1d720b29faf81592f268ba13dc798cb7138ddce5c95c8a65a823f9: Status 404 returned error can't find the container with id 1618af8b7e1d720b29faf81592f268ba13dc798cb7138ddce5c95c8a65a823f9 Jul 22 12:47:56.609564 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:56.609534 2576 generic.go:356] "Generic (PLEG): container finished" podID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerID="0d008323aaf1a8e29b350d2f51c68b2937191d2718b0c99ba7de7d161f9bf9f0" exitCode=0 Jul 22 12:47:56.609903 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:56.609616 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" event={"ID":"aa7d0a0a-964f-4ede-ab83-83abdb9a9525","Type":"ContainerDied","Data":"0d008323aaf1a8e29b350d2f51c68b2937191d2718b0c99ba7de7d161f9bf9f0"} Jul 22 12:47:56.609903 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:56.609671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" event={"ID":"aa7d0a0a-964f-4ede-ab83-83abdb9a9525","Type":"ContainerStarted","Data":"1618af8b7e1d720b29faf81592f268ba13dc798cb7138ddce5c95c8a65a823f9"} Jul 22 12:47:57.622120 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:57.622003 2576 generic.go:356] "Generic (PLEG): container finished" podID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerID="ce6b8a370c98c1914dc382d1e01d2af4c583c4aa79c4da274f2b489825436304" exitCode=0 Jul 22 12:47:57.622512 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:57.622116 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" event={"ID":"3f31666c-b4a9-4e7e-b2df-29bf5be874aa","Type":"ContainerDied","Data":"ce6b8a370c98c1914dc382d1e01d2af4c583c4aa79c4da274f2b489825436304"} Jul 22 12:47:57.624072 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:57.624049 2576 generic.go:356] "Generic (PLEG): container finished" podID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerID="12747f35cc44019f2cc4df5adb0d5ce65b99ea40affe5493a2b1cc1b261828ed" exitCode=0 Jul 22 12:47:57.624156 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:57.624129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" event={"ID":"aa7d0a0a-964f-4ede-ab83-83abdb9a9525","Type":"ContainerDied","Data":"12747f35cc44019f2cc4df5adb0d5ce65b99ea40affe5493a2b1cc1b261828ed"} Jul 22 12:47:57.625934 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:57.625915 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerID="e5a3793c8ec1310aae329136fb948b5a2d8d509cf4c466ae5cb8214ebe8f320a" exitCode=0 Jul 22 12:47:57.626047 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:57.625995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" event={"ID":"0d2bfb32-d761-4047-8be9-86e52e4ec9fb","Type":"ContainerDied","Data":"e5a3793c8ec1310aae329136fb948b5a2d8d509cf4c466ae5cb8214ebe8f320a"} Jul 22 12:47:57.627891 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:57.627873 2576 generic.go:356] "Generic (PLEG): container finished" podID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerID="abde54f664dfbf33710a5ffd5c802fec077623e1418c5f48c1989266b1b840e8" exitCode=0 Jul 22 12:47:57.627984 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:57.627920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" event={"ID":"ea53e9ad-7714-494f-b698-e164671c6ab7","Type":"ContainerDied","Data":"abde54f664dfbf33710a5ffd5c802fec077623e1418c5f48c1989266b1b840e8"} Jul 22 12:47:58.634609 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:58.634581 2576 generic.go:356] "Generic (PLEG): container finished" podID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerID="76335c35a0afddbbf7b38392686d26ab686fcfe7291074714e336eebd99450f3" exitCode=0 Jul 22 12:47:58.635060 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:58.634674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" event={"ID":"3f31666c-b4a9-4e7e-b2df-29bf5be874aa","Type":"ContainerDied","Data":"76335c35a0afddbbf7b38392686d26ab686fcfe7291074714e336eebd99450f3"} Jul 22 12:47:58.636587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:58.636564 2576 generic.go:356] "Generic (PLEG): container finished" podID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerID="123b792e90b1924d44f2f4230c1d73bb123fa95466e48d3624aee708aca1e074" exitCode=0 Jul 22 12:47:58.636687 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:58.636642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" event={"ID":"aa7d0a0a-964f-4ede-ab83-83abdb9a9525","Type":"ContainerDied","Data":"123b792e90b1924d44f2f4230c1d73bb123fa95466e48d3624aee708aca1e074"} Jul 22 12:47:58.638552 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:58.638533 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerID="288e682b33b1779c0bb042032bf6a9eb72037307c84840e745e7d0aab1ad88d1" exitCode=0 Jul 22 12:47:58.638656 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:58.638619 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" event={"ID":"0d2bfb32-d761-4047-8be9-86e52e4ec9fb","Type":"ContainerDied","Data":"288e682b33b1779c0bb042032bf6a9eb72037307c84840e745e7d0aab1ad88d1"} Jul 22 12:47:58.640452 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:58.640435 2576 generic.go:356] "Generic (PLEG): container finished" podID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerID="867cb89d352b108815b4c97e395b552e3dc809ffaf6172aed2a17019ddc8c700" exitCode=0 Jul 22 12:47:58.640535 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:58.640457 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" event={"ID":"ea53e9ad-7714-494f-b698-e164671c6ab7","Type":"ContainerDied","Data":"867cb89d352b108815b4c97e395b552e3dc809ffaf6172aed2a17019ddc8c700"} Jul 22 12:47:59.787033 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.786991 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:47:59.823805 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.823678 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:47:59.836869 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.836843 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:47:59.853325 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.853306 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:47:59.896261 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896197 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-bundle\") pod \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " Jul 22 12:47:59.896261 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896239 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9gf6\" (UniqueName: \"kubernetes.io/projected/ea53e9ad-7714-494f-b698-e164671c6ab7-kube-api-access-l9gf6\") pod \"ea53e9ad-7714-494f-b698-e164671c6ab7\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " Jul 22 12:47:59.896445 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896281 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-util\") pod \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " Jul 22 12:47:59.896445 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896297 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-bundle\") pod \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " Jul 22 12:47:59.896445 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896315 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdmbc\" (UniqueName: \"kubernetes.io/projected/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-kube-api-access-mdmbc\") pod \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " Jul 22 12:47:59.896445 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896334 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-bundle\") pod \"ea53e9ad-7714-494f-b698-e164671c6ab7\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " Jul 22 12:47:59.896445 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896405 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-util\") pod \"ea53e9ad-7714-494f-b698-e164671c6ab7\" (UID: \"ea53e9ad-7714-494f-b698-e164671c6ab7\") " Jul 22 12:47:59.896445 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896443 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxrsq\" (UniqueName: \"kubernetes.io/projected/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-kube-api-access-zxrsq\") pod \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\" (UID: \"0d2bfb32-d761-4047-8be9-86e52e4ec9fb\") " Jul 22 12:47:59.896734 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.896466 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-util\") pod \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\" (UID: \"3f31666c-b4a9-4e7e-b2df-29bf5be874aa\") " Jul 22 12:47:59.897107 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.897064 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-bundle" (OuterVolumeSpecName: "bundle") pod "ea53e9ad-7714-494f-b698-e164671c6ab7" (UID: "ea53e9ad-7714-494f-b698-e164671c6ab7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:47:59.897307 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.897110 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-bundle" (OuterVolumeSpecName: "bundle") pod "3f31666c-b4a9-4e7e-b2df-29bf5be874aa" (UID: "3f31666c-b4a9-4e7e-b2df-29bf5be874aa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:47:59.897307 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.897183 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-bundle" (OuterVolumeSpecName: "bundle") pod "0d2bfb32-d761-4047-8be9-86e52e4ec9fb" (UID: "0d2bfb32-d761-4047-8be9-86e52e4ec9fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:47:59.899359 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.899332 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea53e9ad-7714-494f-b698-e164671c6ab7-kube-api-access-l9gf6" (OuterVolumeSpecName: "kube-api-access-l9gf6") pod "ea53e9ad-7714-494f-b698-e164671c6ab7" (UID: "ea53e9ad-7714-494f-b698-e164671c6ab7"). InnerVolumeSpecName "kube-api-access-l9gf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:47:59.899661 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.899637 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-kube-api-access-mdmbc" (OuterVolumeSpecName: "kube-api-access-mdmbc") pod "3f31666c-b4a9-4e7e-b2df-29bf5be874aa" (UID: "3f31666c-b4a9-4e7e-b2df-29bf5be874aa"). InnerVolumeSpecName "kube-api-access-mdmbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:47:59.899817 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.899793 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-kube-api-access-zxrsq" (OuterVolumeSpecName: "kube-api-access-zxrsq") pod "0d2bfb32-d761-4047-8be9-86e52e4ec9fb" (UID: "0d2bfb32-d761-4047-8be9-86e52e4ec9fb"). InnerVolumeSpecName "kube-api-access-zxrsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:47:59.903341 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.903318 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-util" (OuterVolumeSpecName: "util") pod "ea53e9ad-7714-494f-b698-e164671c6ab7" (UID: "ea53e9ad-7714-494f-b698-e164671c6ab7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:47:59.903565 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.903546 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-util" (OuterVolumeSpecName: "util") pod "3f31666c-b4a9-4e7e-b2df-29bf5be874aa" (UID: "3f31666c-b4a9-4e7e-b2df-29bf5be874aa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:47:59.903692 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.903674 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-util" (OuterVolumeSpecName: "util") pod "0d2bfb32-d761-4047-8be9-86e52e4ec9fb" (UID: "0d2bfb32-d761-4047-8be9-86e52e4ec9fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:47:59.996936 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.996910 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-bundle\") pod \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " Jul 22 12:47:59.997120 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.996969 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkddp\" (UniqueName: \"kubernetes.io/projected/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-kube-api-access-pkddp\") pod \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " Jul 22 12:47:59.997120 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997066 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-util\") pod \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\" (UID: \"aa7d0a0a-964f-4ede-ab83-83abdb9a9525\") " Jul 22 12:47:59.997232 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997193 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-util\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997232 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997203 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-bundle\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997232 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997212 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdmbc\" (UniqueName: \"kubernetes.io/projected/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-kube-api-access-mdmbc\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997232 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997221 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-bundle\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997232 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997229 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ea53e9ad-7714-494f-b698-e164671c6ab7-util\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997443 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997237 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrsq\" (UniqueName: \"kubernetes.io/projected/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-kube-api-access-zxrsq\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997443 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997245 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f31666c-b4a9-4e7e-b2df-29bf5be874aa-util\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997443 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997260 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0d2bfb32-d761-4047-8be9-86e52e4ec9fb-bundle\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997443 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997268 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9gf6\" (UniqueName: \"kubernetes.io/projected/ea53e9ad-7714-494f-b698-e164671c6ab7-kube-api-access-l9gf6\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:47:59.997587 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.997503 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-bundle" (OuterVolumeSpecName: "bundle") pod "aa7d0a0a-964f-4ede-ab83-83abdb9a9525" (UID: "aa7d0a0a-964f-4ede-ab83-83abdb9a9525"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:47:59.999161 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:47:59.999139 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-kube-api-access-pkddp" (OuterVolumeSpecName: "kube-api-access-pkddp") pod "aa7d0a0a-964f-4ede-ab83-83abdb9a9525" (UID: "aa7d0a0a-964f-4ede-ab83-83abdb9a9525"). InnerVolumeSpecName "kube-api-access-pkddp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:48:00.001932 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.001907 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-util" (OuterVolumeSpecName: "util") pod "aa7d0a0a-964f-4ede-ab83-83abdb9a9525" (UID: "aa7d0a0a-964f-4ede-ab83-83abdb9a9525"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:48:00.098651 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.098620 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-util\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:48:00.098651 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.098645 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-bundle\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:48:00.098651 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.098655 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkddp\" (UniqueName: \"kubernetes.io/projected/aa7d0a0a-964f-4ede-ab83-83abdb9a9525-kube-api-access-pkddp\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:48:00.652446 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.652420 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" Jul 22 12:48:00.652446 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.652427 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf" event={"ID":"0d2bfb32-d761-4047-8be9-86e52e4ec9fb","Type":"ContainerDied","Data":"ddcf86700f0604046ba267fb1152a4c1f54cd8e794393d7f71bd04f27d329877"} Jul 22 12:48:00.652654 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.652471 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddcf86700f0604046ba267fb1152a4c1f54cd8e794393d7f71bd04f27d329877" Jul 22 12:48:00.654436 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.654418 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" Jul 22 12:48:00.654563 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.654444 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn" event={"ID":"ea53e9ad-7714-494f-b698-e164671c6ab7","Type":"ContainerDied","Data":"026c4c7544ecf58c6cc96624e50d6f1ad044e73b248c140d06fbdd5186530c8d"} Jul 22 12:48:00.654563 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.654482 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="026c4c7544ecf58c6cc96624e50d6f1ad044e73b248c140d06fbdd5186530c8d" Jul 22 12:48:00.656396 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.656380 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" Jul 22 12:48:00.656502 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.656407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv" event={"ID":"3f31666c-b4a9-4e7e-b2df-29bf5be874aa","Type":"ContainerDied","Data":"4c908012bd05a8cb59dfa279741022370d712b3337ab6702fd7ca2aacb76d593"} Jul 22 12:48:00.656502 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.656441 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c908012bd05a8cb59dfa279741022370d712b3337ab6702fd7ca2aacb76d593" Jul 22 12:48:00.664300 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.664027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" event={"ID":"aa7d0a0a-964f-4ede-ab83-83abdb9a9525","Type":"ContainerDied","Data":"1618af8b7e1d720b29faf81592f268ba13dc798cb7138ddce5c95c8a65a823f9"} Jul 22 12:48:00.664300 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.664109 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1618af8b7e1d720b29faf81592f268ba13dc798cb7138ddce5c95c8a65a823f9" Jul 22 12:48:00.664300 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:00.664206 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c" Jul 22 12:48:17.268426 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268352 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2"] Jul 22 12:48:17.268746 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268697 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerName="extract" Jul 22 12:48:17.268746 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268709 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerName="extract" Jul 22 12:48:17.268746 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268721 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerName="extract" Jul 22 12:48:17.268746 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268728 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerName="extract" Jul 22 12:48:17.268746 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268739 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerName="util" Jul 22 12:48:17.268746 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268745 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerName="util" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268750 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerName="util" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268755 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerName="util" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268761 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerName="pull" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268765 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerName="pull" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268774 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerName="pull" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268779 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerName="pull" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268785 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerName="util" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268789 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerName="util" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268794 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerName="extract" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268798 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerName="extract" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268803 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerName="extract" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268809 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerName="extract" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268817 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerName="util" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268822 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerName="util" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268830 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerName="pull" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268834 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerName="pull" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268841 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerName="pull" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268845 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerName="pull" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268891 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d2bfb32-d761-4047-8be9-86e52e4ec9fb" containerName="extract" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268900 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea53e9ad-7714-494f-b698-e164671c6ab7" containerName="extract" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268906 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa7d0a0a-964f-4ede-ab83-83abdb9a9525" containerName="extract" Jul 22 12:48:17.268930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.268913 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f31666c-b4a9-4e7e-b2df-29bf5be874aa" containerName="extract" Jul 22 12:48:17.271694 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.271674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:17.274643 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.274625 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hv48z\"" Jul 22 12:48:17.275151 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.275132 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 12:48:17.276029 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.275998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 12:48:17.358932 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.358910 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2"] Jul 22 12:48:17.434692 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.434662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2f299dc-abd9-4d5a-895d-edc8c9255622-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" (UID: \"d2f299dc-abd9-4d5a-895d-edc8c9255622\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:17.434794 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.434747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7g2v\" (UniqueName: \"kubernetes.io/projected/d2f299dc-abd9-4d5a-895d-edc8c9255622-kube-api-access-b7g2v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" (UID: \"d2f299dc-abd9-4d5a-895d-edc8c9255622\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:17.535178 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.535102 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2f299dc-abd9-4d5a-895d-edc8c9255622-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" (UID: \"d2f299dc-abd9-4d5a-895d-edc8c9255622\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:17.535302 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.535193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7g2v\" (UniqueName: \"kubernetes.io/projected/d2f299dc-abd9-4d5a-895d-edc8c9255622-kube-api-access-b7g2v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" (UID: \"d2f299dc-abd9-4d5a-895d-edc8c9255622\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:17.535464 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.535445 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2f299dc-abd9-4d5a-895d-edc8c9255622-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" (UID: \"d2f299dc-abd9-4d5a-895d-edc8c9255622\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:17.545051 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.545032 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7g2v\" (UniqueName: \"kubernetes.io/projected/d2f299dc-abd9-4d5a-895d-edc8c9255622-kube-api-access-b7g2v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" (UID: \"d2f299dc-abd9-4d5a-895d-edc8c9255622\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:17.580868 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.580846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:17.714377 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.714356 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2"] Jul 22 12:48:17.716919 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:48:17.716895 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f299dc_abd9_4d5a_895d_edc8c9255622.slice/crio-6207c86a061288ed6bfb250160db9b72fb99823b9c8aab2ed57dad4d27160695 WatchSource:0}: Error finding container 6207c86a061288ed6bfb250160db9b72fb99823b9c8aab2ed57dad4d27160695: Status 404 returned error can't find the container with id 6207c86a061288ed6bfb250160db9b72fb99823b9c8aab2ed57dad4d27160695 Jul 22 12:48:17.746655 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:17.746631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" event={"ID":"d2f299dc-abd9-4d5a-895d-edc8c9255622","Type":"ContainerStarted","Data":"6207c86a061288ed6bfb250160db9b72fb99823b9c8aab2ed57dad4d27160695"} Jul 22 12:48:22.780762 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:22.780718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" event={"ID":"d2f299dc-abd9-4d5a-895d-edc8c9255622","Type":"ContainerStarted","Data":"b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5"} Jul 22 12:48:22.781254 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:22.780844 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:22.805961 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:22.805912 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" podStartSLOduration=1.294424657 podStartE2EDuration="5.805897207s" totalImagesPullingTime="4.51147255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:48:22.803354149 +0000 UTC m=+456.608071046" watchObservedRunningTime="2026-07-22 12:48:22.805897207 +0000 UTC m=+456.610614103" Jul 22 12:48:33.788216 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:33.788176 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:36.052434 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.052403 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2"] Jul 22 12:48:36.052837 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.052592 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" podUID="d2f299dc-abd9-4d5a-895d-edc8c9255622" containerName="manager" containerID="cri-o://b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5" gracePeriod=2 Jul 22 12:48:36.064502 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.064477 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2"] Jul 22 12:48:36.102535 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.102505 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2"] Jul 22 12:48:36.102832 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.102811 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f299dc-abd9-4d5a-895d-edc8c9255622" containerName="manager" Jul 22 12:48:36.102832 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.102827 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f299dc-abd9-4d5a-895d-edc8c9255622" containerName="manager" Jul 22 12:48:36.103041 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.102923 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f299dc-abd9-4d5a-895d-edc8c9255622" containerName="manager" Jul 22 12:48:36.105963 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.105944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.110254 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:48:36.110212 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" is forbidden: User \"system:node:ip-10-0-137-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-88.ec2.internal' and this object" podUID="d2f299dc-abd9-4d5a-895d-edc8c9255622" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:36.131605 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.131579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2"] Jul 22 12:48:36.185410 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.185373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlt6v\" (UniqueName: \"kubernetes.io/projected/637ff6e2-528e-420e-a024-776759df7ed5-kube-api-access-rlt6v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tv9r2\" (UID: \"637ff6e2-528e-420e-a024-776759df7ed5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.185604 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.185478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/637ff6e2-528e-420e-a024-776759df7ed5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tv9r2\" (UID: \"637ff6e2-528e-420e-a024-776759df7ed5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.286722 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.286692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/637ff6e2-528e-420e-a024-776759df7ed5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tv9r2\" (UID: \"637ff6e2-528e-420e-a024-776759df7ed5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.286841 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.286747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlt6v\" (UniqueName: \"kubernetes.io/projected/637ff6e2-528e-420e-a024-776759df7ed5-kube-api-access-rlt6v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tv9r2\" (UID: \"637ff6e2-528e-420e-a024-776759df7ed5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.287181 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.287161 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/637ff6e2-528e-420e-a024-776759df7ed5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tv9r2\" (UID: \"637ff6e2-528e-420e-a024-776759df7ed5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.295668 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.295646 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:36.298341 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:48:36.298320 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" is forbidden: User \"system:node:ip-10-0-137-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-88.ec2.internal' and this object" podUID="d2f299dc-abd9-4d5a-895d-edc8c9255622" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:36.299553 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.299536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlt6v\" (UniqueName: \"kubernetes.io/projected/637ff6e2-528e-420e-a024-776759df7ed5-kube-api-access-rlt6v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tv9r2\" (UID: \"637ff6e2-528e-420e-a024-776759df7ed5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.387950 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.387863 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7g2v\" (UniqueName: \"kubernetes.io/projected/d2f299dc-abd9-4d5a-895d-edc8c9255622-kube-api-access-b7g2v\") pod \"d2f299dc-abd9-4d5a-895d-edc8c9255622\" (UID: \"d2f299dc-abd9-4d5a-895d-edc8c9255622\") " Jul 22 12:48:36.387950 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.387917 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2f299dc-abd9-4d5a-895d-edc8c9255622-extensions-socket-volume\") pod \"d2f299dc-abd9-4d5a-895d-edc8c9255622\" (UID: \"d2f299dc-abd9-4d5a-895d-edc8c9255622\") " Jul 22 12:48:36.388458 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.388428 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f299dc-abd9-4d5a-895d-edc8c9255622-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d2f299dc-abd9-4d5a-895d-edc8c9255622" (UID: "d2f299dc-abd9-4d5a-895d-edc8c9255622"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:48:36.390094 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.390067 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f299dc-abd9-4d5a-895d-edc8c9255622-kube-api-access-b7g2v" (OuterVolumeSpecName: "kube-api-access-b7g2v") pod "d2f299dc-abd9-4d5a-895d-edc8c9255622" (UID: "d2f299dc-abd9-4d5a-895d-edc8c9255622"). InnerVolumeSpecName "kube-api-access-b7g2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:48:36.434258 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.434227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.488527 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.488500 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7g2v\" (UniqueName: \"kubernetes.io/projected/d2f299dc-abd9-4d5a-895d-edc8c9255622-kube-api-access-b7g2v\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:48:36.488605 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.488535 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2f299dc-abd9-4d5a-895d-edc8c9255622-extensions-socket-volume\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:48:36.619886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.618645 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2"] Jul 22 12:48:36.711673 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:48:36.711642 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tbfl2\" is forbidden: User \"system:node:ip-10-0-137-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-88.ec2.internal' and this object" podUID="d2f299dc-abd9-4d5a-895d-edc8c9255622" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:36.712676 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.712653 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f299dc-abd9-4d5a-895d-edc8c9255622" path="/var/lib/kubelet/pods/d2f299dc-abd9-4d5a-895d-edc8c9255622/volumes" Jul 22 12:48:36.851875 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.851842 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2f299dc-abd9-4d5a-895d-edc8c9255622" containerID="b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5" exitCode=0 Jul 22 12:48:36.852075 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.851906 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tbfl2" Jul 22 12:48:36.852075 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.851937 2576 scope.go:117] "RemoveContainer" containerID="b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5" Jul 22 12:48:36.854047 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.853757 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" event={"ID":"637ff6e2-528e-420e-a024-776759df7ed5","Type":"ContainerStarted","Data":"01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903"} Jul 22 12:48:36.854047 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.853814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" event={"ID":"637ff6e2-528e-420e-a024-776759df7ed5","Type":"ContainerStarted","Data":"194f2c57ee774499789233d695ea11c743206e51a915d1988b8c4c03a3ef291d"} Jul 22 12:48:36.854047 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.853899 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:36.860492 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.860470 2576 scope.go:117] "RemoveContainer" containerID="b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5" Jul 22 12:48:36.860717 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:48:36.860701 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5\": container with ID starting with b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5 not found: ID does not exist" containerID="b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5" Jul 22 12:48:36.860765 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.860724 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5"} err="failed to get container status \"b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5\": rpc error: code = NotFound desc = could not find container \"b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5\": container with ID starting with b6226a6fd32c53cc113ffa79a8a4b702d24896f31ed75b35337446dc237481d5 not found: ID does not exist" Jul 22 12:48:36.893096 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:36.893047 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" podStartSLOduration=0.89303145 podStartE2EDuration="893.03145ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:48:36.891244883 +0000 UTC m=+470.695961780" watchObservedRunningTime="2026-07-22 12:48:36.89303145 +0000 UTC m=+470.697748351" Jul 22 12:48:47.863858 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:47.863826 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:53.092815 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.092781 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2"] Jul 22 12:48:53.093231 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.092997 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" podUID="637ff6e2-528e-420e-a024-776759df7ed5" containerName="manager" containerID="cri-o://01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903" gracePeriod=10 Jul 22 12:48:53.334132 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.334106 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:53.422297 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.422271 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/637ff6e2-528e-420e-a024-776759df7ed5-extensions-socket-volume\") pod \"637ff6e2-528e-420e-a024-776759df7ed5\" (UID: \"637ff6e2-528e-420e-a024-776759df7ed5\") " Jul 22 12:48:53.422425 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.422332 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlt6v\" (UniqueName: \"kubernetes.io/projected/637ff6e2-528e-420e-a024-776759df7ed5-kube-api-access-rlt6v\") pod \"637ff6e2-528e-420e-a024-776759df7ed5\" (UID: \"637ff6e2-528e-420e-a024-776759df7ed5\") " Jul 22 12:48:53.422701 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.422677 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637ff6e2-528e-420e-a024-776759df7ed5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "637ff6e2-528e-420e-a024-776759df7ed5" (UID: "637ff6e2-528e-420e-a024-776759df7ed5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:48:53.424452 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.424429 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637ff6e2-528e-420e-a024-776759df7ed5-kube-api-access-rlt6v" (OuterVolumeSpecName: "kube-api-access-rlt6v") pod "637ff6e2-528e-420e-a024-776759df7ed5" (UID: "637ff6e2-528e-420e-a024-776759df7ed5"). InnerVolumeSpecName "kube-api-access-rlt6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:48:53.523112 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.523085 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlt6v\" (UniqueName: \"kubernetes.io/projected/637ff6e2-528e-420e-a024-776759df7ed5-kube-api-access-rlt6v\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:48:53.523112 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.523109 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/637ff6e2-528e-420e-a024-776759df7ed5-extensions-socket-volume\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:48:53.938453 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.938412 2576 generic.go:356] "Generic (PLEG): container finished" podID="637ff6e2-528e-420e-a024-776759df7ed5" containerID="01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903" exitCode=0 Jul 22 12:48:53.938603 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.938457 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" Jul 22 12:48:53.938603 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.938458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" event={"ID":"637ff6e2-528e-420e-a024-776759df7ed5","Type":"ContainerDied","Data":"01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903"} Jul 22 12:48:53.938603 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.938577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2" event={"ID":"637ff6e2-528e-420e-a024-776759df7ed5","Type":"ContainerDied","Data":"194f2c57ee774499789233d695ea11c743206e51a915d1988b8c4c03a3ef291d"} Jul 22 12:48:53.938603 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.938599 2576 scope.go:117] "RemoveContainer" containerID="01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903" Jul 22 12:48:53.947241 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.947225 2576 scope.go:117] "RemoveContainer" containerID="01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903" Jul 22 12:48:53.947475 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:48:53.947459 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903\": container with ID starting with 01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903 not found: ID does not exist" containerID="01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903" Jul 22 12:48:53.947531 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.947483 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903"} err="failed to get container status \"01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903\": rpc error: code = NotFound desc = could not find container \"01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903\": container with ID starting with 01a2b2f1ae525dfd1e9d0866835c09718a64ac45052fa36e68cb0f75a5de0903 not found: ID does not exist" Jul 22 12:48:53.983301 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:53.983273 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2"] Jul 22 12:48:54.007767 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:54.007746 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tv9r2"] Jul 22 12:48:54.712843 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:48:54.712812 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637ff6e2-528e-420e-a024-776759df7ed5" path="/var/lib/kubelet/pods/637ff6e2-528e-420e-a024-776759df7ed5/volumes" Jul 22 12:49:11.707626 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.707593 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4wjdh"] Jul 22 12:49:11.708113 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.707899 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637ff6e2-528e-420e-a024-776759df7ed5" containerName="manager" Jul 22 12:49:11.708113 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.707909 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="637ff6e2-528e-420e-a024-776759df7ed5" containerName="manager" Jul 22 12:49:11.708113 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.707967 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="637ff6e2-528e-420e-a024-776759df7ed5" containerName="manager" Jul 22 12:49:11.710855 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.710840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:11.713538 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.713518 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 12:49:11.714061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.714046 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g556c\"" Jul 22 12:49:11.714177 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.714085 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 22 12:49:11.714177 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.714095 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 12:49:11.720202 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.720185 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4wjdh"] Jul 22 12:49:11.761759 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.761736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8449b720-5a05-4b3f-9d5f-dc875118fce7-config-file\") pod \"limitador-limitador-64c94fd64f-4wjdh\" (UID: \"8449b720-5a05-4b3f-9d5f-dc875118fce7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:11.761902 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.761772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphnp\" (UniqueName: \"kubernetes.io/projected/8449b720-5a05-4b3f-9d5f-dc875118fce7-kube-api-access-bphnp\") pod \"limitador-limitador-64c94fd64f-4wjdh\" (UID: \"8449b720-5a05-4b3f-9d5f-dc875118fce7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:11.802615 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.802585 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4wjdh"] Jul 22 12:49:11.862442 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.862413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphnp\" (UniqueName: \"kubernetes.io/projected/8449b720-5a05-4b3f-9d5f-dc875118fce7-kube-api-access-bphnp\") pod \"limitador-limitador-64c94fd64f-4wjdh\" (UID: \"8449b720-5a05-4b3f-9d5f-dc875118fce7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:11.862588 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.862475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8449b720-5a05-4b3f-9d5f-dc875118fce7-config-file\") pod \"limitador-limitador-64c94fd64f-4wjdh\" (UID: \"8449b720-5a05-4b3f-9d5f-dc875118fce7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:11.863058 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.863040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8449b720-5a05-4b3f-9d5f-dc875118fce7-config-file\") pod \"limitador-limitador-64c94fd64f-4wjdh\" (UID: \"8449b720-5a05-4b3f-9d5f-dc875118fce7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:11.871812 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:11.871791 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphnp\" (UniqueName: \"kubernetes.io/projected/8449b720-5a05-4b3f-9d5f-dc875118fce7-kube-api-access-bphnp\") pod \"limitador-limitador-64c94fd64f-4wjdh\" (UID: \"8449b720-5a05-4b3f-9d5f-dc875118fce7\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:12.020637 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.020578 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:12.125251 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.125222 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:49:12.129866 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.129841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:12.139699 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.139675 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:49:12.153141 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.153082 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4wjdh"] Jul 22 12:49:12.156703 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:49:12.156679 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8449b720_5a05_4b3f_9d5f_dc875118fce7.slice/crio-f61a52a1859af70af3e7306b37fd3f54de2f27605a7ac4c668bf64095660683c WatchSource:0}: Error finding container f61a52a1859af70af3e7306b37fd3f54de2f27605a7ac4c668bf64095660683c: Status 404 returned error can't find the container with id f61a52a1859af70af3e7306b37fd3f54de2f27605a7ac4c668bf64095660683c Jul 22 12:49:12.184059 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.184022 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:49:12.264094 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.264067 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/300abdfe-4403-45c2-b7e1-2a3316aa96d6-config-file\") pod \"limitador-limitador-54d6d988d5-k57ld\" (UID: \"300abdfe-4403-45c2-b7e1-2a3316aa96d6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:12.264250 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.264102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24wx\" (UniqueName: \"kubernetes.io/projected/300abdfe-4403-45c2-b7e1-2a3316aa96d6-kube-api-access-s24wx\") pod \"limitador-limitador-54d6d988d5-k57ld\" (UID: \"300abdfe-4403-45c2-b7e1-2a3316aa96d6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:12.364833 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.364776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/300abdfe-4403-45c2-b7e1-2a3316aa96d6-config-file\") pod \"limitador-limitador-54d6d988d5-k57ld\" (UID: \"300abdfe-4403-45c2-b7e1-2a3316aa96d6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:12.364833 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.364804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24wx\" (UniqueName: \"kubernetes.io/projected/300abdfe-4403-45c2-b7e1-2a3316aa96d6-kube-api-access-s24wx\") pod \"limitador-limitador-54d6d988d5-k57ld\" (UID: \"300abdfe-4403-45c2-b7e1-2a3316aa96d6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:12.365381 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.365367 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/300abdfe-4403-45c2-b7e1-2a3316aa96d6-config-file\") pod \"limitador-limitador-54d6d988d5-k57ld\" (UID: \"300abdfe-4403-45c2-b7e1-2a3316aa96d6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:12.374387 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.374363 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24wx\" (UniqueName: \"kubernetes.io/projected/300abdfe-4403-45c2-b7e1-2a3316aa96d6-kube-api-access-s24wx\") pod \"limitador-limitador-54d6d988d5-k57ld\" (UID: \"300abdfe-4403-45c2-b7e1-2a3316aa96d6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:12.441351 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.441331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:12.574371 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:12.574346 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:49:12.574977 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:49:12.574952 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300abdfe_4403_45c2_b7e1_2a3316aa96d6.slice/crio-07c5956d44d6a284c3fe0984a2a4758083b94272c5a6b30970ca4e6762b3c38b WatchSource:0}: Error finding container 07c5956d44d6a284c3fe0984a2a4758083b94272c5a6b30970ca4e6762b3c38b: Status 404 returned error can't find the container with id 07c5956d44d6a284c3fe0984a2a4758083b94272c5a6b30970ca4e6762b3c38b Jul 22 12:49:13.031489 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:13.031452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" event={"ID":"8449b720-5a05-4b3f-9d5f-dc875118fce7","Type":"ContainerStarted","Data":"f61a52a1859af70af3e7306b37fd3f54de2f27605a7ac4c668bf64095660683c"} Jul 22 12:49:13.032679 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:13.032654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" event={"ID":"300abdfe-4403-45c2-b7e1-2a3316aa96d6","Type":"ContainerStarted","Data":"07c5956d44d6a284c3fe0984a2a4758083b94272c5a6b30970ca4e6762b3c38b"} Jul 22 12:49:16.052737 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:16.052701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" event={"ID":"8449b720-5a05-4b3f-9d5f-dc875118fce7","Type":"ContainerStarted","Data":"c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8"} Jul 22 12:49:16.053222 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:16.052861 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:16.054233 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:16.054209 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" event={"ID":"300abdfe-4403-45c2-b7e1-2a3316aa96d6","Type":"ContainerStarted","Data":"ee330b3418b17f55cd7e2c29bcd2affa81bca20c3e5a567e28f8aeaaa6ebd07f"} Jul 22 12:49:16.054358 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:16.054345 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:16.071769 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:16.071727 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" podStartSLOduration=1.870289322 podStartE2EDuration="5.071715254s" totalImagesPullingTime="3.201425932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:16.071320284 +0000 UTC m=+509.876037182" watchObservedRunningTime="2026-07-22 12:49:16.071715254 +0000 UTC m=+509.876432149" Jul 22 12:49:16.093268 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:16.093213 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" podStartSLOduration=1.321413419 podStartE2EDuration="4.093199558s" totalImagesPullingTime="2.771786139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:16.09144356 +0000 UTC m=+509.896160456" watchObservedRunningTime="2026-07-22 12:49:16.093199558 +0000 UTC m=+509.897916453" Jul 22 12:49:27.060088 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:27.060059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:27.060476 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:27.060131 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-k57ld" Jul 22 12:49:27.131029 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:27.130987 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4wjdh"] Jul 22 12:49:27.131220 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:27.131198 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" podUID="8449b720-5a05-4b3f-9d5f-dc875118fce7" containerName="limitador" containerID="cri-o://c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8" gracePeriod=30 Jul 22 12:49:28.066606 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.066580 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:28.113994 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.113955 2576 generic.go:356] "Generic (PLEG): container finished" podID="8449b720-5a05-4b3f-9d5f-dc875118fce7" containerID="c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8" exitCode=0 Jul 22 12:49:28.113994 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.113993 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" Jul 22 12:49:28.114194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.114040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" event={"ID":"8449b720-5a05-4b3f-9d5f-dc875118fce7","Type":"ContainerDied","Data":"c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8"} Jul 22 12:49:28.114194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.114097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-4wjdh" event={"ID":"8449b720-5a05-4b3f-9d5f-dc875118fce7","Type":"ContainerDied","Data":"f61a52a1859af70af3e7306b37fd3f54de2f27605a7ac4c668bf64095660683c"} Jul 22 12:49:28.114194 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.114127 2576 scope.go:117] "RemoveContainer" containerID="c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8" Jul 22 12:49:28.122391 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.122373 2576 scope.go:117] "RemoveContainer" containerID="c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8" Jul 22 12:49:28.122683 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:49:28.122665 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8\": container with ID starting with c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8 not found: ID does not exist" containerID="c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8" Jul 22 12:49:28.122746 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.122694 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8"} err="failed to get container status \"c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8\": rpc error: code = NotFound desc = could not find container \"c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8\": container with ID starting with c993ea196660958bb7604a40c4900d249e2ff0ee1251a2e26bc0441e29644fb8 not found: ID does not exist" Jul 22 12:49:28.194218 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.194127 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8449b720-5a05-4b3f-9d5f-dc875118fce7-config-file\") pod \"8449b720-5a05-4b3f-9d5f-dc875118fce7\" (UID: \"8449b720-5a05-4b3f-9d5f-dc875118fce7\") " Jul 22 12:49:28.194361 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.194250 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bphnp\" (UniqueName: \"kubernetes.io/projected/8449b720-5a05-4b3f-9d5f-dc875118fce7-kube-api-access-bphnp\") pod \"8449b720-5a05-4b3f-9d5f-dc875118fce7\" (UID: \"8449b720-5a05-4b3f-9d5f-dc875118fce7\") " Jul 22 12:49:28.194513 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.194488 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8449b720-5a05-4b3f-9d5f-dc875118fce7-config-file" (OuterVolumeSpecName: "config-file") pod "8449b720-5a05-4b3f-9d5f-dc875118fce7" (UID: "8449b720-5a05-4b3f-9d5f-dc875118fce7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:49:28.196784 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.196758 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8449b720-5a05-4b3f-9d5f-dc875118fce7-kube-api-access-bphnp" (OuterVolumeSpecName: "kube-api-access-bphnp") pod "8449b720-5a05-4b3f-9d5f-dc875118fce7" (UID: "8449b720-5a05-4b3f-9d5f-dc875118fce7"). InnerVolumeSpecName "kube-api-access-bphnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:49:28.294929 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.294905 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bphnp\" (UniqueName: \"kubernetes.io/projected/8449b720-5a05-4b3f-9d5f-dc875118fce7-kube-api-access-bphnp\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:49:28.294929 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.294929 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8449b720-5a05-4b3f-9d5f-dc875118fce7-config-file\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:49:28.443388 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.438753 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4wjdh"] Jul 22 12:49:28.443388 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.443205 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-4wjdh"] Jul 22 12:49:28.712847 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:28.712812 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8449b720-5a05-4b3f-9d5f-dc875118fce7" path="/var/lib/kubelet/pods/8449b720-5a05-4b3f-9d5f-dc875118fce7/volumes" Jul 22 12:49:52.720782 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.720750 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d56b7d756-6lg6r"] Jul 22 12:49:52.721169 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.721081 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8449b720-5a05-4b3f-9d5f-dc875118fce7" containerName="limitador" Jul 22 12:49:52.721169 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.721093 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8449b720-5a05-4b3f-9d5f-dc875118fce7" containerName="limitador" Jul 22 12:49:52.721169 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.721144 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8449b720-5a05-4b3f-9d5f-dc875118fce7" containerName="limitador" Jul 22 12:49:52.729300 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.729272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:52.732616 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.732589 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 12:49:52.732738 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.732589 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 12:49:52.732738 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.732677 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v9lgc\"" Jul 22 12:49:52.734857 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.734836 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d56b7d756-6lg6r"] Jul 22 12:49:52.787484 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.787462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5169b80-c8bb-4e91-a854-1339515884f1-webhook-cert\") pod \"maas-controller-6d56b7d756-6lg6r\" (UID: \"c5169b80-c8bb-4e91-a854-1339515884f1\") " pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:52.787605 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.787562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72ch8\" (UniqueName: \"kubernetes.io/projected/c5169b80-c8bb-4e91-a854-1339515884f1-kube-api-access-72ch8\") pod \"maas-controller-6d56b7d756-6lg6r\" (UID: \"c5169b80-c8bb-4e91-a854-1339515884f1\") " pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:52.888147 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.888109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5169b80-c8bb-4e91-a854-1339515884f1-webhook-cert\") pod \"maas-controller-6d56b7d756-6lg6r\" (UID: \"c5169b80-c8bb-4e91-a854-1339515884f1\") " pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:52.888323 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.888206 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72ch8\" (UniqueName: \"kubernetes.io/projected/c5169b80-c8bb-4e91-a854-1339515884f1-kube-api-access-72ch8\") pod \"maas-controller-6d56b7d756-6lg6r\" (UID: \"c5169b80-c8bb-4e91-a854-1339515884f1\") " pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:52.890621 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.890600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5169b80-c8bb-4e91-a854-1339515884f1-webhook-cert\") pod \"maas-controller-6d56b7d756-6lg6r\" (UID: \"c5169b80-c8bb-4e91-a854-1339515884f1\") " pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:52.899452 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.899425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72ch8\" (UniqueName: \"kubernetes.io/projected/c5169b80-c8bb-4e91-a854-1339515884f1-kube-api-access-72ch8\") pod \"maas-controller-6d56b7d756-6lg6r\" (UID: \"c5169b80-c8bb-4e91-a854-1339515884f1\") " pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:52.918478 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.918452 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-c4x2g"] Jul 22 12:49:52.922244 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.922228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:52.931160 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.931141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-c4x2g"] Jul 22 12:49:52.988620 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.988545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtfd7\" (UniqueName: \"kubernetes.io/projected/e024efea-831b-4915-8366-e599023e37a1-kube-api-access-gtfd7\") pod \"maas-controller-6c5545ff5f-c4x2g\" (UID: \"e024efea-831b-4915-8366-e599023e37a1\") " pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:52.988749 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:52.988619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e024efea-831b-4915-8366-e599023e37a1-webhook-cert\") pod \"maas-controller-6c5545ff5f-c4x2g\" (UID: \"e024efea-831b-4915-8366-e599023e37a1\") " pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:53.042158 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.042132 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:53.089352 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.089320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e024efea-831b-4915-8366-e599023e37a1-webhook-cert\") pod \"maas-controller-6c5545ff5f-c4x2g\" (UID: \"e024efea-831b-4915-8366-e599023e37a1\") " pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:53.089482 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.089388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtfd7\" (UniqueName: \"kubernetes.io/projected/e024efea-831b-4915-8366-e599023e37a1-kube-api-access-gtfd7\") pod \"maas-controller-6c5545ff5f-c4x2g\" (UID: \"e024efea-831b-4915-8366-e599023e37a1\") " pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:53.093377 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.092368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e024efea-831b-4915-8366-e599023e37a1-webhook-cert\") pod \"maas-controller-6c5545ff5f-c4x2g\" (UID: \"e024efea-831b-4915-8366-e599023e37a1\") " pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:53.099446 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.099373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtfd7\" (UniqueName: \"kubernetes.io/projected/e024efea-831b-4915-8366-e599023e37a1-kube-api-access-gtfd7\") pod \"maas-controller-6c5545ff5f-c4x2g\" (UID: \"e024efea-831b-4915-8366-e599023e37a1\") " pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:53.175939 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.175914 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d56b7d756-6lg6r"] Jul 22 12:49:53.177798 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:49:53.177766 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5169b80_c8bb_4e91_a854_1339515884f1.slice/crio-5db70df08fa0d935609ba2e06526056c68f1644b8cebb530d65bea71de767eec WatchSource:0}: Error finding container 5db70df08fa0d935609ba2e06526056c68f1644b8cebb530d65bea71de767eec: Status 404 returned error can't find the container with id 5db70df08fa0d935609ba2e06526056c68f1644b8cebb530d65bea71de767eec Jul 22 12:49:53.232787 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.232754 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" event={"ID":"c5169b80-c8bb-4e91-a854-1339515884f1","Type":"ContainerStarted","Data":"5db70df08fa0d935609ba2e06526056c68f1644b8cebb530d65bea71de767eec"} Jul 22 12:49:53.235955 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.235933 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:53.369172 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:53.369151 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-c4x2g"] Jul 22 12:49:53.370346 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:49:53.370323 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode024efea_831b_4915_8366_e599023e37a1.slice/crio-4673c9340912c86ec686bf7ff0a993b37279010b4126459b6e1c5106e0d07a9f WatchSource:0}: Error finding container 4673c9340912c86ec686bf7ff0a993b37279010b4126459b6e1c5106e0d07a9f: Status 404 returned error can't find the container with id 4673c9340912c86ec686bf7ff0a993b37279010b4126459b6e1c5106e0d07a9f Jul 22 12:49:54.242142 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:54.242106 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" event={"ID":"e024efea-831b-4915-8366-e599023e37a1","Type":"ContainerStarted","Data":"4673c9340912c86ec686bf7ff0a993b37279010b4126459b6e1c5106e0d07a9f"} Jul 22 12:49:56.254907 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:56.254874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" event={"ID":"c5169b80-c8bb-4e91-a854-1339515884f1","Type":"ContainerStarted","Data":"93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c"} Jul 22 12:49:56.255371 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:56.255049 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:49:56.256702 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:56.256680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" event={"ID":"e024efea-831b-4915-8366-e599023e37a1","Type":"ContainerStarted","Data":"a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70"} Jul 22 12:49:56.256814 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:56.256745 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:49:56.277902 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:56.277863 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" podStartSLOduration=1.627176282 podStartE2EDuration="4.277849693s" totalImagesPullingTime="2.650673411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:56.276929169 +0000 UTC m=+550.081646076" watchObservedRunningTime="2026-07-22 12:49:56.277849693 +0000 UTC m=+550.082566630" Jul 22 12:49:56.302478 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:49:56.302427 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" podStartSLOduration=1.850890942 podStartE2EDuration="4.302414401s" totalImagesPullingTime="2.451523459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:56.301196243 +0000 UTC m=+550.105913140" watchObservedRunningTime="2026-07-22 12:49:56.302414401 +0000 UTC m=+550.107131346" Jul 22 12:50:07.264268 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.264240 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:50:07.264732 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.264317 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:50:07.332846 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.332818 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d56b7d756-6lg6r"] Jul 22 12:50:07.333038 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.332997 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" podUID="c5169b80-c8bb-4e91-a854-1339515884f1" containerName="manager" containerID="cri-o://93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c" gracePeriod=30 Jul 22 12:50:07.578954 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.578927 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:50:07.597084 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.597062 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5169b80-c8bb-4e91-a854-1339515884f1-webhook-cert\") pod \"c5169b80-c8bb-4e91-a854-1339515884f1\" (UID: \"c5169b80-c8bb-4e91-a854-1339515884f1\") " Jul 22 12:50:07.597187 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.597091 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72ch8\" (UniqueName: \"kubernetes.io/projected/c5169b80-c8bb-4e91-a854-1339515884f1-kube-api-access-72ch8\") pod \"c5169b80-c8bb-4e91-a854-1339515884f1\" (UID: \"c5169b80-c8bb-4e91-a854-1339515884f1\") " Jul 22 12:50:07.599174 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.599141 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5169b80-c8bb-4e91-a854-1339515884f1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c5169b80-c8bb-4e91-a854-1339515884f1" (UID: "c5169b80-c8bb-4e91-a854-1339515884f1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:50:07.599258 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.599198 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5169b80-c8bb-4e91-a854-1339515884f1-kube-api-access-72ch8" (OuterVolumeSpecName: "kube-api-access-72ch8") pod "c5169b80-c8bb-4e91-a854-1339515884f1" (UID: "c5169b80-c8bb-4e91-a854-1339515884f1"). InnerVolumeSpecName "kube-api-access-72ch8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:50:07.698111 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.698083 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5169b80-c8bb-4e91-a854-1339515884f1-webhook-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:50:07.698111 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:07.698107 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72ch8\" (UniqueName: \"kubernetes.io/projected/c5169b80-c8bb-4e91-a854-1339515884f1-kube-api-access-72ch8\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:50:08.316906 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.316876 2576 generic.go:356] "Generic (PLEG): container finished" podID="c5169b80-c8bb-4e91-a854-1339515884f1" containerID="93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c" exitCode=0 Jul 22 12:50:08.317313 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.316926 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" Jul 22 12:50:08.317313 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.316957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" event={"ID":"c5169b80-c8bb-4e91-a854-1339515884f1","Type":"ContainerDied","Data":"93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c"} Jul 22 12:50:08.317313 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.317039 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d56b7d756-6lg6r" event={"ID":"c5169b80-c8bb-4e91-a854-1339515884f1","Type":"ContainerDied","Data":"5db70df08fa0d935609ba2e06526056c68f1644b8cebb530d65bea71de767eec"} Jul 22 12:50:08.317313 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.317061 2576 scope.go:117] "RemoveContainer" containerID="93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c" Jul 22 12:50:08.326128 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.326110 2576 scope.go:117] "RemoveContainer" containerID="93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c" Jul 22 12:50:08.326363 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:50:08.326346 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c\": container with ID starting with 93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c not found: ID does not exist" containerID="93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c" Jul 22 12:50:08.326430 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.326370 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c"} err="failed to get container status \"93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c\": rpc error: code = NotFound desc = could not find container \"93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c\": container with ID starting with 93983d0182785a2968899f74177c762da0b7dbde2d29afe4ee303f25d1ef659c not found: ID does not exist" Jul 22 12:50:08.338995 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.338973 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d56b7d756-6lg6r"] Jul 22 12:50:08.343340 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.343322 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d56b7d756-6lg6r"] Jul 22 12:50:08.713097 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:08.713068 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5169b80-c8bb-4e91-a854-1339515884f1" path="/var/lib/kubelet/pods/c5169b80-c8bb-4e91-a854-1339515884f1/volumes" Jul 22 12:50:28.375061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.372517 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd"] Jul 22 12:50:28.375061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.373352 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5169b80-c8bb-4e91-a854-1339515884f1" containerName="manager" Jul 22 12:50:28.375061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.373375 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5169b80-c8bb-4e91-a854-1339515884f1" containerName="manager" Jul 22 12:50:28.375061 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.373552 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5169b80-c8bb-4e91-a854-1339515884f1" containerName="manager" Jul 22 12:50:28.380167 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.380143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.383219 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.383194 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 22 12:50:28.383580 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.383549 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5j75t\"" Jul 22 12:50:28.384263 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.384236 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 12:50:28.384444 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.384426 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:50:28.387398 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.387378 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd"] Jul 22 12:50:28.479550 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.479520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-openshift-service-ca\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.479695 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.479561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht8ch\" (UniqueName: \"kubernetes.io/projected/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-kube-api-access-ht8ch\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.479695 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.479590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-maas-api-tls\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.580171 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.580142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-openshift-service-ca\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.580322 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.580186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht8ch\" (UniqueName: \"kubernetes.io/projected/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-kube-api-access-ht8ch\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.580322 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.580222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-maas-api-tls\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.580806 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.580788 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-openshift-service-ca\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.582726 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.582708 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-maas-api-tls\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.595557 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.595533 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht8ch\" (UniqueName: \"kubernetes.io/projected/73983dd7-6c72-4b48-ae6d-9c4a27f737d2-kube-api-access-ht8ch\") pod \"maas-api-6dfb8748db-pv8zd\" (UID: \"73983dd7-6c72-4b48-ae6d-9c4a27f737d2\") " pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.692595 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.692567 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:28.858045 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.858001 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd"] Jul 22 12:50:28.859524 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:50:28.859494 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73983dd7_6c72_4b48_ae6d_9c4a27f737d2.slice/crio-253decd3d8fe1a2a24ea4bfb51adc010ad604933b5b7f257993fb014be994f28 WatchSource:0}: Error finding container 253decd3d8fe1a2a24ea4bfb51adc010ad604933b5b7f257993fb014be994f28: Status 404 returned error can't find the container with id 253decd3d8fe1a2a24ea4bfb51adc010ad604933b5b7f257993fb014be994f28 Jul 22 12:50:28.959028 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.958949 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hggtd"] Jul 22 12:50:28.964490 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.964469 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-wgztj"] Jul 22 12:50:28.964623 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.964597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:28.967951 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.967934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:28.968312 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.968297 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 22 12:50:28.980921 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.980895 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-wgztj"] Jul 22 12:50:28.986485 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:28.986464 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hggtd"] Jul 22 12:50:29.085257 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.085230 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeff253e-7449-44f6-896c-c3172c83cc8d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-wgztj\" (UID: \"aeff253e-7449-44f6-896c-c3172c83cc8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:29.085399 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.085264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6ll\" (UniqueName: \"kubernetes.io/projected/aeff253e-7449-44f6-896c-c3172c83cc8d-kube-api-access-cd6ll\") pod \"payload-processing-5bf47cd988-wgztj\" (UID: \"aeff253e-7449-44f6-896c-c3172c83cc8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:29.085399 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.085323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6154c67f-3cfd-471b-b1c4-847e905ab587-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hggtd\" (UID: \"6154c67f-3cfd-471b-b1c4-847e905ab587\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:29.085399 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.085349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj7mv\" (UniqueName: \"kubernetes.io/projected/6154c67f-3cfd-471b-b1c4-847e905ab587-kube-api-access-wj7mv\") pod \"payload-pre-processing-7677b774bb-hggtd\" (UID: \"6154c67f-3cfd-471b-b1c4-847e905ab587\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:29.109532 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.109504 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:50:29.185909 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.185879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeff253e-7449-44f6-896c-c3172c83cc8d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-wgztj\" (UID: \"aeff253e-7449-44f6-896c-c3172c83cc8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:29.186125 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.185915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6ll\" (UniqueName: \"kubernetes.io/projected/aeff253e-7449-44f6-896c-c3172c83cc8d-kube-api-access-cd6ll\") pod \"payload-processing-5bf47cd988-wgztj\" (UID: \"aeff253e-7449-44f6-896c-c3172c83cc8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:29.186125 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.185991 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6154c67f-3cfd-471b-b1c4-847e905ab587-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hggtd\" (UID: \"6154c67f-3cfd-471b-b1c4-847e905ab587\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:29.186125 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.186052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj7mv\" (UniqueName: \"kubernetes.io/projected/6154c67f-3cfd-471b-b1c4-847e905ab587-kube-api-access-wj7mv\") pod \"payload-pre-processing-7677b774bb-hggtd\" (UID: \"6154c67f-3cfd-471b-b1c4-847e905ab587\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:29.186701 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.186678 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6154c67f-3cfd-471b-b1c4-847e905ab587-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-hggtd\" (UID: \"6154c67f-3cfd-471b-b1c4-847e905ab587\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:29.186807 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.186783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeff253e-7449-44f6-896c-c3172c83cc8d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-wgztj\" (UID: \"aeff253e-7449-44f6-896c-c3172c83cc8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:29.194930 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.194903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj7mv\" (UniqueName: \"kubernetes.io/projected/6154c67f-3cfd-471b-b1c4-847e905ab587-kube-api-access-wj7mv\") pod \"payload-pre-processing-7677b774bb-hggtd\" (UID: \"6154c67f-3cfd-471b-b1c4-847e905ab587\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:29.195087 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.195061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6ll\" (UniqueName: \"kubernetes.io/projected/aeff253e-7449-44f6-896c-c3172c83cc8d-kube-api-access-cd6ll\") pod \"payload-processing-5bf47cd988-wgztj\" (UID: \"aeff253e-7449-44f6-896c-c3172c83cc8d\") " pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:29.275648 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.275562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:29.283186 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.283161 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:29.424466 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.424438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" event={"ID":"73983dd7-6c72-4b48-ae6d-9c4a27f737d2","Type":"ContainerStarted","Data":"253decd3d8fe1a2a24ea4bfb51adc010ad604933b5b7f257993fb014be994f28"} Jul 22 12:50:29.435620 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.435590 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-hggtd"] Jul 22 12:50:29.437662 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:50:29.437635 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6154c67f_3cfd_471b_b1c4_847e905ab587.slice/crio-fd33f02802da173b2758ecc2054bd409b509b0080120aeedca462d4fade9afb2 WatchSource:0}: Error finding container fd33f02802da173b2758ecc2054bd409b509b0080120aeedca462d4fade9afb2: Status 404 returned error can't find the container with id fd33f02802da173b2758ecc2054bd409b509b0080120aeedca462d4fade9afb2 Jul 22 12:50:29.452663 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:29.452643 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-wgztj"] Jul 22 12:50:29.454576 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:50:29.454549 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeff253e_7449_44f6_896c_c3172c83cc8d.slice/crio-68940373b7b2501f1da0968315c6683497f6a59a320238b1b58dff41503b9ce1 WatchSource:0}: Error finding container 68940373b7b2501f1da0968315c6683497f6a59a320238b1b58dff41503b9ce1: Status 404 returned error can't find the container with id 68940373b7b2501f1da0968315c6683497f6a59a320238b1b58dff41503b9ce1 Jul 22 12:50:30.435759 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:30.435723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" event={"ID":"aeff253e-7449-44f6-896c-c3172c83cc8d","Type":"ContainerStarted","Data":"68940373b7b2501f1da0968315c6683497f6a59a320238b1b58dff41503b9ce1"} Jul 22 12:50:30.437719 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:30.437650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" event={"ID":"6154c67f-3cfd-471b-b1c4-847e905ab587","Type":"ContainerStarted","Data":"fd33f02802da173b2758ecc2054bd409b509b0080120aeedca462d4fade9afb2"} Jul 22 12:50:31.444027 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.443974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" event={"ID":"73983dd7-6c72-4b48-ae6d-9c4a27f737d2","Type":"ContainerStarted","Data":"77aeac7e6ebce4b5bb3d5be95008b6b68b2c2cdfa8a49b6a9fd50d27014b76c2"} Jul 22 12:50:31.444451 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.444181 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:31.445524 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.445500 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" event={"ID":"aeff253e-7449-44f6-896c-c3172c83cc8d","Type":"ContainerStarted","Data":"569ad3944b990cbd268fbdb2713b8aa1ec0f8b36980e415b0926a9585bd3a743"} Jul 22 12:50:31.445657 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.445540 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:31.446893 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.446872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" event={"ID":"6154c67f-3cfd-471b-b1c4-847e905ab587","Type":"ContainerStarted","Data":"6d208ce3ca96dd70b2449ed151ce91340f456aa47bb7a3e57e588a59bce9ff73"} Jul 22 12:50:31.446987 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.446918 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:31.466649 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.466608 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" podStartSLOduration=1.65884802 podStartE2EDuration="3.466595887s" totalImagesPullingTime="1.807747867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:50:31.464200751 +0000 UTC m=+585.268917645" watchObservedRunningTime="2026-07-22 12:50:31.466595887 +0000 UTC m=+585.271312783" Jul 22 12:50:31.484537 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.484474 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" podStartSLOduration=1.7063587660000001 podStartE2EDuration="3.484462783s" totalImagesPullingTime="1.778104017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:50:31.483435868 +0000 UTC m=+585.288152767" watchObservedRunningTime="2026-07-22 12:50:31.484462783 +0000 UTC m=+585.289179677" Jul 22 12:50:31.501743 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:31.501711 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" podStartSLOduration=1.7124923220000001 podStartE2EDuration="3.501700588s" totalImagesPullingTime="1.789208266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:50:31.500863148 +0000 UTC m=+585.305580047" watchObservedRunningTime="2026-07-22 12:50:31.501700588 +0000 UTC m=+585.306417484" Jul 22 12:50:37.457774 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:37.457742 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6dfb8748db-pv8zd" Jul 22 12:50:42.454679 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:42.454652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-wgztj" Jul 22 12:50:42.455084 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:42.454805 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-hggtd" Jul 22 12:50:46.606517 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:46.606492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:50:46.606942 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:46.606545 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:50:46.613624 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:46.613604 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:50:46.613881 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:46.613867 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:50:48.906886 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:48.906852 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:50:52.999112 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:50:52.999077 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:51:13.787641 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.787573 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz"] Jul 22 12:51:13.790832 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.790816 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.794348 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.794325 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 12:51:13.794348 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.794331 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nqcsn\"" Jul 22 12:51:13.794524 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.794331 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 22 12:51:13.794524 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.794520 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 12:51:13.801159 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.801138 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz"] Jul 22 12:51:13.851508 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.851476 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.851663 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.851536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.851663 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.851607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.851663 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.851630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.851800 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.851665 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.851800 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.851688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552r2\" (UniqueName: \"kubernetes.io/projected/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-kube-api-access-552r2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.952828 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.952801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.952989 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.952856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.952989 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.952874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.952989 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.952900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.952989 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.952926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552r2\" (UniqueName: \"kubernetes.io/projected/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-kube-api-access-552r2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.952989 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.952960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.953268 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.953253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.953342 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.953324 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.953398 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.953333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.955580 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.955558 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.955780 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.955744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:13.962346 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:13.962319 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-552r2\" (UniqueName: \"kubernetes.io/projected/4fcd1130-cef4-49c6-ba35-36728bd2bf1a-kube-api-access-552r2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nntfz\" (UID: \"4fcd1130-cef4-49c6-ba35-36728bd2bf1a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:14.101933 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:14.101857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:14.244880 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:14.244853 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz"] Jul 22 12:51:14.246272 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:51:14.246245 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcd1130_cef4_49c6_ba35_36728bd2bf1a.slice/crio-bd673eb9b33737584c4ac8bc1a5e82bce5e9a5cba07846bca37a26fd7505e25c WatchSource:0}: Error finding container bd673eb9b33737584c4ac8bc1a5e82bce5e9a5cba07846bca37a26fd7505e25c: Status 404 returned error can't find the container with id bd673eb9b33737584c4ac8bc1a5e82bce5e9a5cba07846bca37a26fd7505e25c Jul 22 12:51:14.247599 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:14.247580 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:51:14.662928 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:14.662888 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" event={"ID":"4fcd1130-cef4-49c6-ba35-36728bd2bf1a","Type":"ContainerStarted","Data":"bd673eb9b33737584c4ac8bc1a5e82bce5e9a5cba07846bca37a26fd7505e25c"} Jul 22 12:51:15.001086 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:15.000910 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:51:15.670696 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:15.670655 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" event={"ID":"4fcd1130-cef4-49c6-ba35-36728bd2bf1a","Type":"ContainerStarted","Data":"33cc308a9d33b8a56981590254d04563de84bfe0c4a0335e5f40fdcd957405d5"} Jul 22 12:51:15.691459 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:15.691415 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" podStartSLOduration=1.592699686 podStartE2EDuration="2.691400076s" totalImagesPullingTime="1.09870039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:51:15.690528937 +0000 UTC m=+629.495245845" watchObservedRunningTime="2026-07-22 12:51:15.691400076 +0000 UTC m=+629.496116969" Jul 22 12:51:24.102181 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:24.102147 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:24.102611 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:24.102200 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:24.114341 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:24.114322 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:24.728115 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:24.728089 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nntfz" Jul 22 12:51:37.644372 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:37.644335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:51:46.527840 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:46.527806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:51:57.098571 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:51:57.098541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:53:15.693137 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:15.693106 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:53:28.910894 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:28.910858 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:53:34.440344 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.440307 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-c4x2g"] Jul 22 12:53:34.440803 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.440544 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" podUID="e024efea-831b-4915-8366-e599023e37a1" containerName="manager" containerID="cri-o://a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70" gracePeriod=30 Jul 22 12:53:34.688591 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.688568 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:53:34.817191 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.817119 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e024efea-831b-4915-8366-e599023e37a1-webhook-cert\") pod \"e024efea-831b-4915-8366-e599023e37a1\" (UID: \"e024efea-831b-4915-8366-e599023e37a1\") " Jul 22 12:53:34.817191 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.817161 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtfd7\" (UniqueName: \"kubernetes.io/projected/e024efea-831b-4915-8366-e599023e37a1-kube-api-access-gtfd7\") pod \"e024efea-831b-4915-8366-e599023e37a1\" (UID: \"e024efea-831b-4915-8366-e599023e37a1\") " Jul 22 12:53:34.819309 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.819278 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e024efea-831b-4915-8366-e599023e37a1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e024efea-831b-4915-8366-e599023e37a1" (UID: "e024efea-831b-4915-8366-e599023e37a1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:53:34.819433 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.819367 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e024efea-831b-4915-8366-e599023e37a1-kube-api-access-gtfd7" (OuterVolumeSpecName: "kube-api-access-gtfd7") pod "e024efea-831b-4915-8366-e599023e37a1" (UID: "e024efea-831b-4915-8366-e599023e37a1"). InnerVolumeSpecName "kube-api-access-gtfd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:53:34.918572 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.918546 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e024efea-831b-4915-8366-e599023e37a1-webhook-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:53:34.918572 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:34.918568 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtfd7\" (UniqueName: \"kubernetes.io/projected/e024efea-831b-4915-8366-e599023e37a1-kube-api-access-gtfd7\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:53:35.378002 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.377961 2576 generic.go:356] "Generic (PLEG): container finished" podID="e024efea-831b-4915-8366-e599023e37a1" containerID="a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70" exitCode=0 Jul 22 12:53:35.378174 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.378007 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" Jul 22 12:53:35.378174 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.378053 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" event={"ID":"e024efea-831b-4915-8366-e599023e37a1","Type":"ContainerDied","Data":"a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70"} Jul 22 12:53:35.378174 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.378115 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-c4x2g" event={"ID":"e024efea-831b-4915-8366-e599023e37a1","Type":"ContainerDied","Data":"4673c9340912c86ec686bf7ff0a993b37279010b4126459b6e1c5106e0d07a9f"} Jul 22 12:53:35.378174 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.378144 2576 scope.go:117] "RemoveContainer" containerID="a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70" Jul 22 12:53:35.387781 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.387766 2576 scope.go:117] "RemoveContainer" containerID="a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70" Jul 22 12:53:35.388068 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:53:35.388051 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70\": container with ID starting with a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70 not found: ID does not exist" containerID="a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70" Jul 22 12:53:35.388119 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.388077 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70"} err="failed to get container status \"a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70\": rpc error: code = NotFound desc = could not find container \"a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70\": container with ID starting with a30e21dc9d2e8bcb22b12085a2910ccbdf107bfda2a820bf23e6fe1ea442cc70 not found: ID does not exist" Jul 22 12:53:35.401922 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.401899 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-c4x2g"] Jul 22 12:53:35.414221 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:35.414201 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-c4x2g"] Jul 22 12:53:36.713984 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:36.713944 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e024efea-831b-4915-8366-e599023e37a1" path="/var/lib/kubelet/pods/e024efea-831b-4915-8366-e599023e37a1/volumes" Jul 22 12:53:38.001214 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:53:38.001174 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:54:36.282038 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.281993 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-qghlx"] Jul 22 12:54:36.282575 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.282479 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e024efea-831b-4915-8366-e599023e37a1" containerName="manager" Jul 22 12:54:36.282575 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.282495 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e024efea-831b-4915-8366-e599023e37a1" containerName="manager" Jul 22 12:54:36.282675 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.282597 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e024efea-831b-4915-8366-e599023e37a1" containerName="manager" Jul 22 12:54:36.285724 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.285709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:36.289586 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.289563 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 12:54:36.289586 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.289563 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v9lgc\"" Jul 22 12:54:36.289750 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.289571 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 12:54:36.297519 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.297496 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-qghlx"] Jul 22 12:54:36.418182 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.418131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mnpr\" (UniqueName: \"kubernetes.io/projected/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-kube-api-access-8mnpr\") pod \"maas-controller-6c5545ff5f-qghlx\" (UID: \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\") " pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:36.418368 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.418258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-webhook-cert\") pod \"maas-controller-6c5545ff5f-qghlx\" (UID: \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\") " pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:36.518877 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.518843 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mnpr\" (UniqueName: \"kubernetes.io/projected/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-kube-api-access-8mnpr\") pod \"maas-controller-6c5545ff5f-qghlx\" (UID: \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\") " pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:36.519060 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.518898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-webhook-cert\") pod \"maas-controller-6c5545ff5f-qghlx\" (UID: \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\") " pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:36.521658 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.521622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-webhook-cert\") pod \"maas-controller-6c5545ff5f-qghlx\" (UID: \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\") " pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:36.529075 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.529052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mnpr\" (UniqueName: \"kubernetes.io/projected/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-kube-api-access-8mnpr\") pod \"maas-controller-6c5545ff5f-qghlx\" (UID: \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\") " pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:36.596724 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.596629 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:36.737728 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:36.737705 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-qghlx"] Jul 22 12:54:36.740320 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:54:36.740292 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe20a4d1_b236_4fae_8f29_c10f4f78ccd8.slice/crio-a7539611c710290f5bca80ee8c4c8ed44ca92b3901cc25f57ccab3ea8955fc03 WatchSource:0}: Error finding container a7539611c710290f5bca80ee8c4c8ed44ca92b3901cc25f57ccab3ea8955fc03: Status 404 returned error can't find the container with id a7539611c710290f5bca80ee8c4c8ed44ca92b3901cc25f57ccab3ea8955fc03 Jul 22 12:54:37.692992 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:37.692960 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" event={"ID":"be20a4d1-b236-4fae-8f29-c10f4f78ccd8","Type":"ContainerStarted","Data":"be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00"} Jul 22 12:54:37.693438 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:37.693007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" event={"ID":"be20a4d1-b236-4fae-8f29-c10f4f78ccd8","Type":"ContainerStarted","Data":"a7539611c710290f5bca80ee8c4c8ed44ca92b3901cc25f57ccab3ea8955fc03"} Jul 22 12:54:37.693438 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:37.693150 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:37.714450 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:37.714405 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" podStartSLOduration=1.109250406 podStartE2EDuration="1.714390113s" totalImagesPullingTime="605.139707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:54:37.713424502 +0000 UTC m=+831.518141399" watchObservedRunningTime="2026-07-22 12:54:37.714390113 +0000 UTC m=+831.519107008" Jul 22 12:54:48.700278 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:48.700251 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:53.375389 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.375356 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-qghlx"] Jul 22 12:54:53.375784 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.375543 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" podUID="be20a4d1-b236-4fae-8f29-c10f4f78ccd8" containerName="manager" containerID="cri-o://be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00" gracePeriod=30 Jul 22 12:54:53.622383 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.622360 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:53.644308 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.644249 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-webhook-cert\") pod \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\" (UID: \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\") " Jul 22 12:54:53.644423 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.644352 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mnpr\" (UniqueName: \"kubernetes.io/projected/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-kube-api-access-8mnpr\") pod \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\" (UID: \"be20a4d1-b236-4fae-8f29-c10f4f78ccd8\") " Jul 22 12:54:53.646913 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.646880 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "be20a4d1-b236-4fae-8f29-c10f4f78ccd8" (UID: "be20a4d1-b236-4fae-8f29-c10f4f78ccd8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:54:53.646913 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.646895 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-kube-api-access-8mnpr" (OuterVolumeSpecName: "kube-api-access-8mnpr") pod "be20a4d1-b236-4fae-8f29-c10f4f78ccd8" (UID: "be20a4d1-b236-4fae-8f29-c10f4f78ccd8"). InnerVolumeSpecName "kube-api-access-8mnpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:54:53.744942 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.744915 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-webhook-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:54:53.744942 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.744935 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mnpr\" (UniqueName: \"kubernetes.io/projected/be20a4d1-b236-4fae-8f29-c10f4f78ccd8-kube-api-access-8mnpr\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 12:54:53.779680 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.779649 2576 generic.go:356] "Generic (PLEG): container finished" podID="be20a4d1-b236-4fae-8f29-c10f4f78ccd8" containerID="be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00" exitCode=0 Jul 22 12:54:53.779775 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.779691 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" Jul 22 12:54:53.779775 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.779739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" event={"ID":"be20a4d1-b236-4fae-8f29-c10f4f78ccd8","Type":"ContainerDied","Data":"be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00"} Jul 22 12:54:53.779845 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.779785 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-qghlx" event={"ID":"be20a4d1-b236-4fae-8f29-c10f4f78ccd8","Type":"ContainerDied","Data":"a7539611c710290f5bca80ee8c4c8ed44ca92b3901cc25f57ccab3ea8955fc03"} Jul 22 12:54:53.779845 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.779811 2576 scope.go:117] "RemoveContainer" containerID="be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00" Jul 22 12:54:53.788730 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.788699 2576 scope.go:117] "RemoveContainer" containerID="be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00" Jul 22 12:54:53.788972 ip-10-0-137-88 kubenswrapper[2576]: E0722 12:54:53.788946 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00\": container with ID starting with be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00 not found: ID does not exist" containerID="be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00" Jul 22 12:54:53.789040 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.788980 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00"} err="failed to get container status \"be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00\": rpc error: code = NotFound desc = could not find container \"be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00\": container with ID starting with be9b8ac2a9a796cdbd77986c71dca8db87afb10eaf1ee4986c4ea9dcbd857f00 not found: ID does not exist" Jul 22 12:54:53.803630 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.803603 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-qghlx"] Jul 22 12:54:53.806664 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:53.806647 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-qghlx"] Jul 22 12:54:54.640813 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.640780 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-wfzpv"] Jul 22 12:54:54.641201 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.641180 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be20a4d1-b236-4fae-8f29-c10f4f78ccd8" containerName="manager" Jul 22 12:54:54.641201 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.641191 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="be20a4d1-b236-4fae-8f29-c10f4f78ccd8" containerName="manager" Jul 22 12:54:54.641269 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.641258 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="be20a4d1-b236-4fae-8f29-c10f4f78ccd8" containerName="manager" Jul 22 12:54:54.645627 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.645608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:54.648262 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.648242 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 22 12:54:54.649231 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.649197 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-v9lgc\"" Jul 22 12:54:54.649231 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.649215 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 22 12:54:54.650299 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.650283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92e2671d-18e7-49dd-9dee-faf7fbcd6888-webhook-cert\") pod \"maas-controller-6c5545ff5f-wfzpv\" (UID: \"92e2671d-18e7-49dd-9dee-faf7fbcd6888\") " pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:54.650390 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.650377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vskwz\" (UniqueName: \"kubernetes.io/projected/92e2671d-18e7-49dd-9dee-faf7fbcd6888-kube-api-access-vskwz\") pod \"maas-controller-6c5545ff5f-wfzpv\" (UID: \"92e2671d-18e7-49dd-9dee-faf7fbcd6888\") " pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:54.655283 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.655261 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-wfzpv"] Jul 22 12:54:54.712306 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.712282 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be20a4d1-b236-4fae-8f29-c10f4f78ccd8" path="/var/lib/kubelet/pods/be20a4d1-b236-4fae-8f29-c10f4f78ccd8/volumes" Jul 22 12:54:54.751578 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.751553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vskwz\" (UniqueName: \"kubernetes.io/projected/92e2671d-18e7-49dd-9dee-faf7fbcd6888-kube-api-access-vskwz\") pod \"maas-controller-6c5545ff5f-wfzpv\" (UID: \"92e2671d-18e7-49dd-9dee-faf7fbcd6888\") " pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:54.751826 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.751805 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92e2671d-18e7-49dd-9dee-faf7fbcd6888-webhook-cert\") pod \"maas-controller-6c5545ff5f-wfzpv\" (UID: \"92e2671d-18e7-49dd-9dee-faf7fbcd6888\") " pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:54.754280 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.754254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92e2671d-18e7-49dd-9dee-faf7fbcd6888-webhook-cert\") pod \"maas-controller-6c5545ff5f-wfzpv\" (UID: \"92e2671d-18e7-49dd-9dee-faf7fbcd6888\") " pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:54.760871 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.760851 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vskwz\" (UniqueName: \"kubernetes.io/projected/92e2671d-18e7-49dd-9dee-faf7fbcd6888-kube-api-access-vskwz\") pod \"maas-controller-6c5545ff5f-wfzpv\" (UID: \"92e2671d-18e7-49dd-9dee-faf7fbcd6888\") " pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:54.956683 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:54.956654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:55.087826 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:55.087799 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c5545ff5f-wfzpv"] Jul 22 12:54:55.089279 ip-10-0-137-88 kubenswrapper[2576]: W0722 12:54:55.089253 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e2671d_18e7_49dd_9dee_faf7fbcd6888.slice/crio-bc4de5453d6c14168e24dca4842bfea8ff569f1067d89fa5d922ecedacde6dbc WatchSource:0}: Error finding container bc4de5453d6c14168e24dca4842bfea8ff569f1067d89fa5d922ecedacde6dbc: Status 404 returned error can't find the container with id bc4de5453d6c14168e24dca4842bfea8ff569f1067d89fa5d922ecedacde6dbc Jul 22 12:54:55.792361 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:55.792329 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" event={"ID":"92e2671d-18e7-49dd-9dee-faf7fbcd6888","Type":"ContainerStarted","Data":"e6e2cb8ba48968f067767f1e8a18af32f91968a72813064b52f657a778a9ef24"} Jul 22 12:54:55.792749 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:55.792378 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" event={"ID":"92e2671d-18e7-49dd-9dee-faf7fbcd6888","Type":"ContainerStarted","Data":"bc4de5453d6c14168e24dca4842bfea8ff569f1067d89fa5d922ecedacde6dbc"} Jul 22 12:54:55.792749 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:55.792409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:54:55.839333 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:54:55.839286 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" podStartSLOduration=1.322498331 podStartE2EDuration="1.83927068s" totalImagesPullingTime="516.772349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:54:55.83689482 +0000 UTC m=+849.641611715" watchObservedRunningTime="2026-07-22 12:54:55.83927068 +0000 UTC m=+849.643987577" Jul 22 12:55:06.799657 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:55:06.799630 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c5545ff5f-wfzpv" Jul 22 12:55:15.200120 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:55:15.200085 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:55:24.213938 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:55:24.213901 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:55:46.638944 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:55:46.638916 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:55:46.640897 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:55:46.640879 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 12:55:46.646055 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:55:46.646030 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:55:46.647951 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:55:46.647932 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 12:55:56.999609 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:55:56.999575 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:56:07.638247 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:56:07.638216 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:56:16.106619 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:56:16.106583 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:56:25.604963 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:56:25.604932 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:56:35.101846 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:56:35.101811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:56:45.692820 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:56:45.692788 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:56:58.814541 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:56:58.814509 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:57:07.197439 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:57:07.197360 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:57:29.592324 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:57:29.592287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:57:38.901885 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:57:38.901852 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:58:02.814074 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:58:02.814035 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:58:14.389851 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:58:14.389813 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:58:29.295088 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:58:29.295051 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:58:39.498063 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:58:39.497964 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:58:51.392216 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:58:51.392183 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:59:00.194521 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:59:00.194489 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 12:59:09.295601 ip-10-0-137-88 kubenswrapper[2576]: I0722 12:59:09.295572 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:00:46.675246 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:00:46.675165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:00:46.677497 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:00:46.677477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:00:46.682168 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:00:46.682148 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:00:46.684444 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:00:46.684428 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:00:56.504340 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:00:56.504301 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:01:07.795958 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:01:07.795922 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:01:17.601329 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:01:17.601297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:01:29.212191 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:01:29.212157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:01:41.499831 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:01:41.499743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:01:49.308360 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:01:49.308321 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:01:57.317843 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:01:57.317806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:02:06.409646 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:02:06.409605 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:02:15.008541 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:02:15.008507 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:02:25.990230 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:02:25.990188 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:02:37.495302 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:02:37.495268 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:02:44.299142 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:02:44.299097 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:03:03.706679 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:03:03.706649 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:03:13.319569 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:03:13.319478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:03:28.999456 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:03:28.999417 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:03:42.706502 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:03:42.706469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:04:07.210634 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:04:07.210595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:04:21.099811 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:04:21.099776 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:04:28.295850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:04:28.295812 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:04:37.719720 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:04:37.719627 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:04:45.807382 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:04:45.807348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:04:55.499072 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:04:55.499029 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:05:04.813256 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:04.813213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:05:15.506793 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:15.506759 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:05:25.005477 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:25.005447 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:05:34.402028 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:34.401977 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:05:46.706624 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:46.706597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:05:46.711621 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:46.711597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:05:46.715315 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:46.715293 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:05:46.719425 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:46.719410 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:05:47.695789 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:47.695752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:05:56.800836 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:05:56.800804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:06:06.203305 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:06:06.203266 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:06:17.119523 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:06:17.119491 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:06:29.424519 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:06:29.424479 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:06:36.404790 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:06:36.404755 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:06:51.201640 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:06:51.201604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:07:03.497732 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:03.497689 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:07:16.790488 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:16.790451 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:07:31.001775 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:31.001741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:07:41.294284 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:41.294198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:07:52.193716 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.193666 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9"] Jul 22 13:07:52.198420 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.198390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.202906 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.202882 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e1001375-serving-cert\"" Jul 22 13:07:52.206554 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.206529 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9"] Jul 22 13:07:52.270839 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.270808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.271042 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.270850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52tbv\" (UniqueName: \"kubernetes.io/projected/81fc6776-87b3-4313-8f90-54d01534e762-kube-api-access-52tbv\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.271042 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.270968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81fc6776-87b3-4313-8f90-54d01534e762-openshift-service-ca\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.371721 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.371676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81fc6776-87b3-4313-8f90-54d01534e762-openshift-service-ca\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.371903 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.371743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.371903 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.371769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52tbv\" (UniqueName: \"kubernetes.io/projected/81fc6776-87b3-4313-8f90-54d01534e762-kube-api-access-52tbv\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.371983 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:07:52.371913 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-serving-cert: secret "maas-api-e2e-ait-e1001375-serving-cert" not found Jul 22 13:07:52.372071 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:07:52.372054 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls podName:81fc6776-87b3-4313-8f90-54d01534e762 nodeName:}" failed. No retries permitted until 2026-07-22 13:07:52.871993118 +0000 UTC m=+1626.676710013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls") pod "maas-api-e2e-ait-e1001375-747cffb49f-llkt9" (UID: "81fc6776-87b3-4313-8f90-54d01534e762") : secret "maas-api-e2e-ait-e1001375-serving-cert" not found Jul 22 13:07:52.372505 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.372486 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81fc6776-87b3-4313-8f90-54d01534e762-openshift-service-ca\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.382088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.382055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52tbv\" (UniqueName: \"kubernetes.io/projected/81fc6776-87b3-4313-8f90-54d01534e762-kube-api-access-52tbv\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.876193 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.876148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:52.878952 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:52.878929 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls\") pod \"maas-api-e2e-ait-e1001375-747cffb49f-llkt9\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:53.109982 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:53.109950 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:53.271364 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:53.271341 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9"] Jul 22 13:07:53.273640 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:07:53.273604 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fc6776_87b3_4313_8f90_54d01534e762.slice/crio-d85e126c1816b20371bcf1a05ffef56d944ebeb565430f14254bb395a55cb873 WatchSource:0}: Error finding container d85e126c1816b20371bcf1a05ffef56d944ebeb565430f14254bb395a55cb873: Status 404 returned error can't find the container with id d85e126c1816b20371bcf1a05ffef56d944ebeb565430f14254bb395a55cb873 Jul 22 13:07:53.275196 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:53.275174 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 13:07:53.770256 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:53.770221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" event={"ID":"81fc6776-87b3-4313-8f90-54d01534e762","Type":"ContainerStarted","Data":"d85e126c1816b20371bcf1a05ffef56d944ebeb565430f14254bb395a55cb873"} Jul 22 13:07:54.777382 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:54.777344 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" event={"ID":"81fc6776-87b3-4313-8f90-54d01534e762","Type":"ContainerStarted","Data":"e7db46ca321c3ef7c20aefeadb33d4b394d5e95049747ddd854f5931e8d41353"} Jul 22 13:07:54.777817 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:54.777460 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:07:54.845117 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:54.845050 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" podStartSLOduration=2.1753832060000002 podStartE2EDuration="2.845024615s" totalImagesPullingTime="669.641409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:07:54.825201255 +0000 UTC m=+1628.629918150" watchObservedRunningTime="2026-07-22 13:07:54.845024615 +0000 UTC m=+1628.649741504" Jul 22 13:07:54.846701 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:07:54.846678 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:08:00.788856 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:00.788829 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:08:18.259265 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.259228 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2"] Jul 22 13:08:18.262694 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.262675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" Jul 22 13:08:18.265329 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.265309 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:08:18.271534 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.271510 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2"] Jul 22 13:08:18.305627 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.305590 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e0d10d-4adf-4f5c-a98b-ca0aa0e37316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1001375-ksps2\" (UID: \"67e0d10d-4adf-4f5c-a98b-ca0aa0e37316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" Jul 22 13:08:18.407222 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.407128 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e0d10d-4adf-4f5c-a98b-ca0aa0e37316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1001375-ksps2\" (UID: \"67e0d10d-4adf-4f5c-a98b-ca0aa0e37316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" Jul 22 13:08:18.407732 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.407710 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e0d10d-4adf-4f5c-a98b-ca0aa0e37316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e1001375-ksps2\" (UID: \"67e0d10d-4adf-4f5c-a98b-ca0aa0e37316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" Jul 22 13:08:18.574434 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.574337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" Jul 22 13:08:18.718160 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.718140 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2"] Jul 22 13:08:18.719548 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:08:18.719517 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e0d10d_4adf_4f5c_a98b_ca0aa0e37316.slice/crio-a9fa176a7647cc38df71860589383696caccf4cf6fb3a9b11d27d4eca3e438be WatchSource:0}: Error finding container a9fa176a7647cc38df71860589383696caccf4cf6fb3a9b11d27d4eca3e438be: Status 404 returned error can't find the container with id a9fa176a7647cc38df71860589383696caccf4cf6fb3a9b11d27d4eca3e438be Jul 22 13:08:18.909612 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:18.909574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" event={"ID":"67e0d10d-4adf-4f5c-a98b-ca0aa0e37316","Type":"ContainerStarted","Data":"a9fa176a7647cc38df71860589383696caccf4cf6fb3a9b11d27d4eca3e438be"} Jul 22 13:08:26.955721 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:26.955683 2576 generic.go:356] "Generic (PLEG): container finished" podID="67e0d10d-4adf-4f5c-a98b-ca0aa0e37316" containerID="3860273fd9d97237063f0708ba2815a05e7878d39fd19457ab24eb7e490d5f62" exitCode=0 Jul 22 13:08:26.956223 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:26.955768 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" event={"ID":"67e0d10d-4adf-4f5c-a98b-ca0aa0e37316","Type":"ContainerDied","Data":"3860273fd9d97237063f0708ba2815a05e7878d39fd19457ab24eb7e490d5f62"} Jul 22 13:08:28.094895 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:28.094872 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" Jul 22 13:08:28.194249 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:28.194215 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e0d10d-4adf-4f5c-a98b-ca0aa0e37316-maas-api-service-ca\") pod \"67e0d10d-4adf-4f5c-a98b-ca0aa0e37316\" (UID: \"67e0d10d-4adf-4f5c-a98b-ca0aa0e37316\") " Jul 22 13:08:28.194567 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:28.194545 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e0d10d-4adf-4f5c-a98b-ca0aa0e37316-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "67e0d10d-4adf-4f5c-a98b-ca0aa0e37316" (UID: "67e0d10d-4adf-4f5c-a98b-ca0aa0e37316"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:08:28.295302 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:28.295221 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67e0d10d-4adf-4f5c-a98b-ca0aa0e37316-maas-api-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:08:28.967677 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:28.967643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" event={"ID":"67e0d10d-4adf-4f5c-a98b-ca0aa0e37316","Type":"ContainerDied","Data":"a9fa176a7647cc38df71860589383696caccf4cf6fb3a9b11d27d4eca3e438be"} Jul 22 13:08:28.967677 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:28.967667 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2" Jul 22 13:08:28.967878 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:28.967691 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9fa176a7647cc38df71860589383696caccf4cf6fb3a9b11d27d4eca3e438be" Jul 22 13:08:38.793803 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:38.793757 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9"] Jul 22 13:08:38.794245 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:38.793969 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" podUID="81fc6776-87b3-4313-8f90-54d01534e762" containerName="maas-api" containerID="cri-o://e7db46ca321c3ef7c20aefeadb33d4b394d5e95049747ddd854f5931e8d41353" gracePeriod=30 Jul 22 13:08:39.025224 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.025185 2576 generic.go:356] "Generic (PLEG): container finished" podID="81fc6776-87b3-4313-8f90-54d01534e762" containerID="e7db46ca321c3ef7c20aefeadb33d4b394d5e95049747ddd854f5931e8d41353" exitCode=0 Jul 22 13:08:39.025414 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.025246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" event={"ID":"81fc6776-87b3-4313-8f90-54d01534e762","Type":"ContainerDied","Data":"e7db46ca321c3ef7c20aefeadb33d4b394d5e95049747ddd854f5931e8d41353"} Jul 22 13:08:39.057472 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.057403 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:08:39.192440 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.192401 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls\") pod \"81fc6776-87b3-4313-8f90-54d01534e762\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " Jul 22 13:08:39.192440 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.192447 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81fc6776-87b3-4313-8f90-54d01534e762-openshift-service-ca\") pod \"81fc6776-87b3-4313-8f90-54d01534e762\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " Jul 22 13:08:39.192650 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.192484 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52tbv\" (UniqueName: \"kubernetes.io/projected/81fc6776-87b3-4313-8f90-54d01534e762-kube-api-access-52tbv\") pod \"81fc6776-87b3-4313-8f90-54d01534e762\" (UID: \"81fc6776-87b3-4313-8f90-54d01534e762\") " Jul 22 13:08:39.192830 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.192805 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fc6776-87b3-4313-8f90-54d01534e762-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81fc6776-87b3-4313-8f90-54d01534e762" (UID: "81fc6776-87b3-4313-8f90-54d01534e762"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:08:39.194864 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.194838 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81fc6776-87b3-4313-8f90-54d01534e762" (UID: "81fc6776-87b3-4313-8f90-54d01534e762"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:08:39.194936 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.194842 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fc6776-87b3-4313-8f90-54d01534e762-kube-api-access-52tbv" (OuterVolumeSpecName: "kube-api-access-52tbv") pod "81fc6776-87b3-4313-8f90-54d01534e762" (UID: "81fc6776-87b3-4313-8f90-54d01534e762"). InnerVolumeSpecName "kube-api-access-52tbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:08:39.293622 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.293589 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81fc6776-87b3-4313-8f90-54d01534e762-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:08:39.293622 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.293621 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81fc6776-87b3-4313-8f90-54d01534e762-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:08:39.293622 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:39.293631 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52tbv\" (UniqueName: \"kubernetes.io/projected/81fc6776-87b3-4313-8f90-54d01534e762-kube-api-access-52tbv\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:08:40.032068 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:40.032037 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" Jul 22 13:08:40.032068 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:40.032048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9" event={"ID":"81fc6776-87b3-4313-8f90-54d01534e762","Type":"ContainerDied","Data":"d85e126c1816b20371bcf1a05ffef56d944ebeb565430f14254bb395a55cb873"} Jul 22 13:08:40.032557 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:40.032110 2576 scope.go:117] "RemoveContainer" containerID="e7db46ca321c3ef7c20aefeadb33d4b394d5e95049747ddd854f5931e8d41353" Jul 22 13:08:40.055067 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:40.055006 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9"] Jul 22 13:08:40.059221 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:40.059192 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1001375-747cffb49f-llkt9"] Jul 22 13:08:40.712812 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:40.712773 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81fc6776-87b3-4313-8f90-54d01534e762" path="/var/lib/kubelet/pods/81fc6776-87b3-4313-8f90-54d01534e762/volumes" Jul 22 13:08:41.895189 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:41.895156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:08:43.344989 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:43.344953 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2"] Jul 22 13:08:43.348065 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:43.348036 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e1001375-ksps2"] Jul 22 13:08:44.712594 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:44.712563 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e0d10d-4adf-4f5c-a98b-ca0aa0e37316" path="/var/lib/kubelet/pods/67e0d10d-4adf-4f5c-a98b-ca0aa0e37316/volumes" Jul 22 13:08:47.520910 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.520878 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5"] Jul 22 13:08:47.521293 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.521256 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e0d10d-4adf-4f5c-a98b-ca0aa0e37316" containerName="revoke-keys" Jul 22 13:08:47.521293 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.521268 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e0d10d-4adf-4f5c-a98b-ca0aa0e37316" containerName="revoke-keys" Jul 22 13:08:47.521293 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.521279 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fc6776-87b3-4313-8f90-54d01534e762" containerName="maas-api" Jul 22 13:08:47.521293 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.521285 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fc6776-87b3-4313-8f90-54d01534e762" containerName="maas-api" Jul 22 13:08:47.521425 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.521357 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fc6776-87b3-4313-8f90-54d01534e762" containerName="maas-api" Jul 22 13:08:47.521425 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.521364 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e0d10d-4adf-4f5c-a98b-ca0aa0e37316" containerName="revoke-keys" Jul 22 13:08:47.525595 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.525574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.528229 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.528214 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-879879e7-serving-cert\"" Jul 22 13:08:47.535795 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.535774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5"] Jul 22 13:08:47.667670 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.667626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7sl9\" (UniqueName: \"kubernetes.io/projected/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-kube-api-access-m7sl9\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.667864 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.667711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-maas-api-tls\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.667864 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.667816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-openshift-service-ca\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.768384 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.768347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7sl9\" (UniqueName: \"kubernetes.io/projected/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-kube-api-access-m7sl9\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.768572 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.768392 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-maas-api-tls\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.768572 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.768510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-openshift-service-ca\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.769207 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.769182 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-openshift-service-ca\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.771071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.771025 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-maas-api-tls\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.777164 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.777140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7sl9\" (UniqueName: \"kubernetes.io/projected/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-kube-api-access-m7sl9\") pod \"maas-api-e2e-migrate-879879e7-766487b676-jgkw5\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.837203 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.837160 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:47.990714 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:47.990690 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5"] Jul 22 13:08:47.992556 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:08:47.992526 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfeef27f_a1f7_4077_b0fb_f9880c7e9282.slice/crio-e39844281907ed1557caa7c3ed368de56eb67fa8e30788746c86ff80e97faa2d WatchSource:0}: Error finding container e39844281907ed1557caa7c3ed368de56eb67fa8e30788746c86ff80e97faa2d: Status 404 returned error can't find the container with id e39844281907ed1557caa7c3ed368de56eb67fa8e30788746c86ff80e97faa2d Jul 22 13:08:48.077530 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:48.077443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" event={"ID":"bfeef27f-a1f7-4077-b0fb-f9880c7e9282","Type":"ContainerStarted","Data":"e39844281907ed1557caa7c3ed368de56eb67fa8e30788746c86ff80e97faa2d"} Jul 22 13:08:49.084483 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:49.084439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" event={"ID":"bfeef27f-a1f7-4077-b0fb-f9880c7e9282","Type":"ContainerStarted","Data":"4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd"} Jul 22 13:08:49.084909 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:49.084578 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:08:49.108461 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:49.108391 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" podStartSLOduration=1.567626873 podStartE2EDuration="2.108368223s" totalImagesPullingTime="540.74135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:08:49.107176235 +0000 UTC m=+1682.911893131" watchObservedRunningTime="2026-07-22 13:08:49.108368223 +0000 UTC m=+1682.913085121" Jul 22 13:08:52.013394 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:52.013358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:08:55.096932 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:08:55.096900 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:09:12.492508 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.492436 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf"] Jul 22 13:09:12.497824 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.497807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" Jul 22 13:09:12.500501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.500481 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:09:12.505653 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.505629 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf"] Jul 22 13:09:12.578352 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.578324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ca147-f00b-4f45-846c-7e4eba41e31f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf\" (UID: \"750ca147-f00b-4f45-846c-7e4eba41e31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" Jul 22 13:09:12.679347 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.679322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ca147-f00b-4f45-846c-7e4eba41e31f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf\" (UID: \"750ca147-f00b-4f45-846c-7e4eba41e31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" Jul 22 13:09:12.679937 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.679917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ca147-f00b-4f45-846c-7e4eba41e31f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf\" (UID: \"750ca147-f00b-4f45-846c-7e4eba41e31f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" Jul 22 13:09:12.813181 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.813114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" Jul 22 13:09:12.951199 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:12.951175 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf"] Jul 22 13:09:12.952879 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:09:12.952825 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750ca147_f00b_4f45_846c_7e4eba41e31f.slice/crio-53ad34c83c386e0559b4807c32f2ef60429f1be406718a206edd21ca750f0779 WatchSource:0}: Error finding container 53ad34c83c386e0559b4807c32f2ef60429f1be406718a206edd21ca750f0779: Status 404 returned error can't find the container with id 53ad34c83c386e0559b4807c32f2ef60429f1be406718a206edd21ca750f0779 Jul 22 13:09:13.219301 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:13.219262 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" event={"ID":"750ca147-f00b-4f45-846c-7e4eba41e31f","Type":"ContainerStarted","Data":"7c47c7f84df60de64ffde2b2e5bae6edbff259046bb1995f4d0ab90acc13e621"} Jul 22 13:09:13.219428 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:13.219320 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" event={"ID":"750ca147-f00b-4f45-846c-7e4eba41e31f","Type":"ContainerStarted","Data":"53ad34c83c386e0559b4807c32f2ef60429f1be406718a206edd21ca750f0779"} Jul 22 13:09:13.238428 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:13.238381 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" podStartSLOduration=1.238364691 podStartE2EDuration="1.238364691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:09:13.235787115 +0000 UTC m=+1707.040504022" watchObservedRunningTime="2026-07-22 13:09:13.238364691 +0000 UTC m=+1707.043081628" Jul 22 13:09:14.226453 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:14.226420 2576 generic.go:356] "Generic (PLEG): container finished" podID="750ca147-f00b-4f45-846c-7e4eba41e31f" containerID="7c47c7f84df60de64ffde2b2e5bae6edbff259046bb1995f4d0ab90acc13e621" exitCode=0 Jul 22 13:09:14.226866 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:14.226499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" event={"ID":"750ca147-f00b-4f45-846c-7e4eba41e31f","Type":"ContainerDied","Data":"7c47c7f84df60de64ffde2b2e5bae6edbff259046bb1995f4d0ab90acc13e621"} Jul 22 13:09:15.374190 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:15.374165 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" Jul 22 13:09:15.403531 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:15.403497 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ca147-f00b-4f45-846c-7e4eba41e31f-maas-api-service-ca\") pod \"750ca147-f00b-4f45-846c-7e4eba41e31f\" (UID: \"750ca147-f00b-4f45-846c-7e4eba41e31f\") " Jul 22 13:09:15.403877 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:15.403852 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750ca147-f00b-4f45-846c-7e4eba41e31f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "750ca147-f00b-4f45-846c-7e4eba41e31f" (UID: "750ca147-f00b-4f45-846c-7e4eba41e31f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:09:15.505187 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:15.505113 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750ca147-f00b-4f45-846c-7e4eba41e31f-maas-api-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:09:16.240763 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:16.240730 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" Jul 22 13:09:16.240926 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:16.240758 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf" event={"ID":"750ca147-f00b-4f45-846c-7e4eba41e31f","Type":"ContainerDied","Data":"53ad34c83c386e0559b4807c32f2ef60429f1be406718a206edd21ca750f0779"} Jul 22 13:09:16.240926 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:16.240831 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53ad34c83c386e0559b4807c32f2ef60429f1be406718a206edd21ca750f0779" Jul 22 13:09:23.262196 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.262163 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5"] Jul 22 13:09:23.262608 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.262370 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" podUID="bfeef27f-a1f7-4077-b0fb-f9880c7e9282" containerName="maas-api" containerID="cri-o://4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd" gracePeriod=30 Jul 22 13:09:23.531996 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.531975 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:09:23.570778 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.570755 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7sl9\" (UniqueName: \"kubernetes.io/projected/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-kube-api-access-m7sl9\") pod \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " Jul 22 13:09:23.570926 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.570822 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-openshift-service-ca\") pod \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " Jul 22 13:09:23.570996 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.570937 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-maas-api-tls\") pod \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\" (UID: \"bfeef27f-a1f7-4077-b0fb-f9880c7e9282\") " Jul 22 13:09:23.571166 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.571141 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bfeef27f-a1f7-4077-b0fb-f9880c7e9282" (UID: "bfeef27f-a1f7-4077-b0fb-f9880c7e9282"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:09:23.571280 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.571269 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:09:23.572976 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.572952 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-kube-api-access-m7sl9" (OuterVolumeSpecName: "kube-api-access-m7sl9") pod "bfeef27f-a1f7-4077-b0fb-f9880c7e9282" (UID: "bfeef27f-a1f7-4077-b0fb-f9880c7e9282"). InnerVolumeSpecName "kube-api-access-m7sl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:09:23.573136 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.573116 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bfeef27f-a1f7-4077-b0fb-f9880c7e9282" (UID: "bfeef27f-a1f7-4077-b0fb-f9880c7e9282"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:09:23.672400 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.672368 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:09:23.672400 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:23.672392 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7sl9\" (UniqueName: \"kubernetes.io/projected/bfeef27f-a1f7-4077-b0fb-f9880c7e9282-kube-api-access-m7sl9\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:09:24.284408 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.284374 2576 generic.go:356] "Generic (PLEG): container finished" podID="bfeef27f-a1f7-4077-b0fb-f9880c7e9282" containerID="4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd" exitCode=0 Jul 22 13:09:24.284862 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.284416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" event={"ID":"bfeef27f-a1f7-4077-b0fb-f9880c7e9282","Type":"ContainerDied","Data":"4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd"} Jul 22 13:09:24.284862 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.284424 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" Jul 22 13:09:24.284862 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.284457 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5" event={"ID":"bfeef27f-a1f7-4077-b0fb-f9880c7e9282","Type":"ContainerDied","Data":"e39844281907ed1557caa7c3ed368de56eb67fa8e30788746c86ff80e97faa2d"} Jul 22 13:09:24.284862 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.284480 2576 scope.go:117] "RemoveContainer" containerID="4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd" Jul 22 13:09:24.294385 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.294367 2576 scope.go:117] "RemoveContainer" containerID="4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd" Jul 22 13:09:24.294866 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:09:24.294800 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd\": container with ID starting with 4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd not found: ID does not exist" containerID="4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd" Jul 22 13:09:24.294866 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.294838 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd"} err="failed to get container status \"4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd\": rpc error: code = NotFound desc = could not find container \"4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd\": container with ID starting with 4803a6ed3fd408db42930cdf425d30b96c023b70bd8fe8f11028dd6930215dbd not found: ID does not exist" Jul 22 13:09:24.316348 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.316305 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5"] Jul 22 13:09:24.318849 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.318824 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-879879e7-766487b676-jgkw5"] Jul 22 13:09:24.713269 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:24.713225 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfeef27f-a1f7-4077-b0fb-f9880c7e9282" path="/var/lib/kubelet/pods/bfeef27f-a1f7-4077-b0fb-f9880c7e9282/volumes" Jul 22 13:09:26.206920 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:26.206883 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:09:27.647623 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:27.647589 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf"] Jul 22 13:09:27.654628 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:27.654598 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-879879e7-bv6nf"] Jul 22 13:09:28.714817 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.714775 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="750ca147-f00b-4f45-846c-7e4eba41e31f" path="/var/lib/kubelet/pods/750ca147-f00b-4f45-846c-7e4eba41e31f/volumes" Jul 22 13:09:28.867902 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.867867 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph"] Jul 22 13:09:28.868257 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.868243 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfeef27f-a1f7-4077-b0fb-f9880c7e9282" containerName="maas-api" Jul 22 13:09:28.868310 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.868258 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfeef27f-a1f7-4077-b0fb-f9880c7e9282" containerName="maas-api" Jul 22 13:09:28.868310 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.868274 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="750ca147-f00b-4f45-846c-7e4eba41e31f" containerName="revoke-keys" Jul 22 13:09:28.868310 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.868279 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="750ca147-f00b-4f45-846c-7e4eba41e31f" containerName="revoke-keys" Jul 22 13:09:28.868405 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.868348 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfeef27f-a1f7-4077-b0fb-f9880c7e9282" containerName="maas-api" Jul 22 13:09:28.868405 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.868357 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="750ca147-f00b-4f45-846c-7e4eba41e31f" containerName="revoke-keys" Jul 22 13:09:28.872722 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.872706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:28.876735 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.876698 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e0df68f3-openshift-default-dockercfg-44xcm\"" Jul 22 13:09:28.900715 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:28.900686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph"] Jul 22 13:09:29.020473 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637e437-eb09-4258-95dc-6e1ec75b879f-istiod-ca-cert\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.020473 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020429 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-podinfo\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.020473 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-socket\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.020708 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020493 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-envoy\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.020708 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgc2\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-kube-api-access-jqgc2\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.020708 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-certs\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.020708 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-token\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.020708 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-data\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.020708 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.020664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-credential-socket\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.121973 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.121939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-podinfo\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122169 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.121984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-socket\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122169 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-envoy\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122169 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgc2\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-kube-api-access-jqgc2\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122169 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122097 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-certs\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122169 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-token\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122169 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-data\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122481 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-credential-socket\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122481 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637e437-eb09-4258-95dc-6e1ec75b879f-istiod-ca-cert\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122481 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122407 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-socket\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122631 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-data\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122694 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-certs\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.122751 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.122704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-credential-socket\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.123043 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.123001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637e437-eb09-4258-95dc-6e1ec75b879f-istiod-ca-cert\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.124479 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.124463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-envoy\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.124763 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.124746 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-podinfo\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.131302 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.131275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-token\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.131501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.131479 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgc2\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-kube-api-access-jqgc2\") pod \"e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.182742 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.182706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:29.344732 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:29.344703 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph"] Jul 22 13:09:29.347659 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:09:29.347629 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8637e437_eb09_4258_95dc_6e1ec75b879f.slice/crio-78f30aff463c4847d812f66809ecbc70fd746911b0857007b6c8244fd64d5321 WatchSource:0}: Error finding container 78f30aff463c4847d812f66809ecbc70fd746911b0857007b6c8244fd64d5321: Status 404 returned error can't find the container with id 78f30aff463c4847d812f66809ecbc70fd746911b0857007b6c8244fd64d5321 Jul 22 13:09:30.324627 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:30.324592 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" event={"ID":"8637e437-eb09-4258-95dc-6e1ec75b879f","Type":"ContainerStarted","Data":"78f30aff463c4847d812f66809ecbc70fd746911b0857007b6c8244fd64d5321"} Jul 22 13:09:32.212349 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:32.212313 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:09:32.212659 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:32.212389 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:09:32.212659 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:32.212417 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:09:32.342139 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:32.342104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" event={"ID":"8637e437-eb09-4258-95dc-6e1ec75b879f","Type":"ContainerStarted","Data":"f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941"} Jul 22 13:09:32.365997 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:32.365951 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" podStartSLOduration=1.504049018 podStartE2EDuration="4.365937531s" totalImagesPullingTime="2.861888513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:09:32.364738646 +0000 UTC m=+1726.169455569" watchObservedRunningTime="2026-07-22 13:09:32.365937531 +0000 UTC m=+1726.170654427" Jul 22 13:09:33.183224 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.183190 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:33.187591 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.187568 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:33.347687 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.347663 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:33.348569 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.348550 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:33.853971 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.853940 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh"] Jul 22 13:09:33.857897 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.857872 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:33.861457 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.861431 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e0df68f3-serving-cert\"" Jul 22 13:09:33.874656 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.874624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh"] Jul 22 13:09:33.968573 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.968540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824fv\" (UniqueName: \"kubernetes.io/projected/901e16e6-76b0-426c-8905-f3306a1d16b0-kube-api-access-824fv\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:33.968744 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.968622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:33.968744 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:33.968649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901e16e6-76b0-426c-8905-f3306a1d16b0-openshift-service-ca\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.069694 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.069664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.069694 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.069695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901e16e6-76b0-426c-8905-f3306a1d16b0-openshift-service-ca\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.069869 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.069741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824fv\" (UniqueName: \"kubernetes.io/projected/901e16e6-76b0-426c-8905-f3306a1d16b0-kube-api-access-824fv\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.069869 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:09:34.069812 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-serving-cert: secret "maas-api-e2e-ait-e0df68f3-serving-cert" not found Jul 22 13:09:34.069945 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:09:34.069873 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls podName:901e16e6-76b0-426c-8905-f3306a1d16b0 nodeName:}" failed. No retries permitted until 2026-07-22 13:09:34.569857574 +0000 UTC m=+1728.374574449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls") pod "maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" (UID: "901e16e6-76b0-426c-8905-f3306a1d16b0") : secret "maas-api-e2e-ait-e0df68f3-serving-cert" not found Jul 22 13:09:34.070391 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.070373 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901e16e6-76b0-426c-8905-f3306a1d16b0-openshift-service-ca\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.093462 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.093435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824fv\" (UniqueName: \"kubernetes.io/projected/901e16e6-76b0-426c-8905-f3306a1d16b0-kube-api-access-824fv\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.573636 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.573593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.576215 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.576193 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls\") pod \"maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.769299 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.769267 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:34.921477 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:34.921452 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh"] Jul 22 13:09:34.922920 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:09:34.922898 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901e16e6_76b0_426c_8905_f3306a1d16b0.slice/crio-b4f27f62131c3c2752c9a55a4edef4c408ab1950221ff63da90463aeb73d6f22 WatchSource:0}: Error finding container b4f27f62131c3c2752c9a55a4edef4c408ab1950221ff63da90463aeb73d6f22: Status 404 returned error can't find the container with id b4f27f62131c3c2752c9a55a4edef4c408ab1950221ff63da90463aeb73d6f22 Jul 22 13:09:35.361075 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:35.361040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" event={"ID":"901e16e6-76b0-426c-8905-f3306a1d16b0","Type":"ContainerStarted","Data":"b4f27f62131c3c2752c9a55a4edef4c408ab1950221ff63da90463aeb73d6f22"} Jul 22 13:09:37.372817 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:37.372780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" event={"ID":"901e16e6-76b0-426c-8905-f3306a1d16b0","Type":"ContainerStarted","Data":"052e66024c8942d4aa51cb2e12908fde466a276dab3bff9290569bf3324f9d6f"} Jul 22 13:09:37.373240 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:37.372900 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:37.394571 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:37.394530 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" podStartSLOduration=2.480737333 podStartE2EDuration="4.394517299s" totalImagesPullingTime="1.913779966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:09:37.392138816 +0000 UTC m=+1731.196855729" watchObservedRunningTime="2026-07-22 13:09:37.394517299 +0000 UTC m=+1731.199234174" Jul 22 13:09:38.216260 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:38.216230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:09:43.383679 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:43.383652 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:09:46.434640 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:46.434609 2576 generic.go:356] "Generic (PLEG): container finished" podID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerID="f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941" exitCode=0 Jul 22 13:09:46.435181 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:46.434687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" event={"ID":"8637e437-eb09-4258-95dc-6e1ec75b879f","Type":"ContainerDied","Data":"f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941"} Jul 22 13:09:46.435253 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:46.435197 2576 scope.go:117] "RemoveContainer" containerID="f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941" Jul 22 13:09:46.437202 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:46.437176 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:09:46.437290 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:46.437242 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:09:46.437290 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:46.437283 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:09:47.441537 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:47.441497 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" event={"ID":"8637e437-eb09-4258-95dc-6e1ec75b879f","Type":"ContainerStarted","Data":"236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7"} Jul 22 13:09:48.183979 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:48.183942 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:48.185091 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:48.185069 2576 patch_prober.go:28] interesting pod/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 22 13:09:48.185154 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:48.185119 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 22 13:09:48.348312 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:48.348282 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:49.183547 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:49.183507 2576 patch_prober.go:28] interesting pod/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 22 13:09:49.183941 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:49.183581 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 22 13:09:50.183525 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:50.183494 2576 patch_prober.go:28] interesting pod/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 22 13:09:50.183699 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:50.183541 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 22 13:09:51.183636 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:51.183597 2576 patch_prober.go:28] interesting pod/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 22 13:09:51.183818 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:51.183658 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 22 13:09:52.187314 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:52.187287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:09:52.188436 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:09:52.188416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:10:10.385273 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.385223 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh"] Jul 22 13:10:10.385751 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.385532 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" podUID="901e16e6-76b0-426c-8905-f3306a1d16b0" containerName="maas-api" containerID="cri-o://052e66024c8942d4aa51cb2e12908fde466a276dab3bff9290569bf3324f9d6f" gracePeriod=30 Jul 22 13:10:10.565190 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.565134 2576 generic.go:356] "Generic (PLEG): container finished" podID="901e16e6-76b0-426c-8905-f3306a1d16b0" containerID="052e66024c8942d4aa51cb2e12908fde466a276dab3bff9290569bf3324f9d6f" exitCode=0 Jul 22 13:10:10.565346 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.565282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" event={"ID":"901e16e6-76b0-426c-8905-f3306a1d16b0","Type":"ContainerDied","Data":"052e66024c8942d4aa51cb2e12908fde466a276dab3bff9290569bf3324f9d6f"} Jul 22 13:10:10.650136 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.650058 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:10:10.706425 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.706389 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901e16e6-76b0-426c-8905-f3306a1d16b0-openshift-service-ca\") pod \"901e16e6-76b0-426c-8905-f3306a1d16b0\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " Jul 22 13:10:10.706628 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.706464 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-824fv\" (UniqueName: \"kubernetes.io/projected/901e16e6-76b0-426c-8905-f3306a1d16b0-kube-api-access-824fv\") pod \"901e16e6-76b0-426c-8905-f3306a1d16b0\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " Jul 22 13:10:10.706628 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.706491 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls\") pod \"901e16e6-76b0-426c-8905-f3306a1d16b0\" (UID: \"901e16e6-76b0-426c-8905-f3306a1d16b0\") " Jul 22 13:10:10.706888 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.706854 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901e16e6-76b0-426c-8905-f3306a1d16b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "901e16e6-76b0-426c-8905-f3306a1d16b0" (UID: "901e16e6-76b0-426c-8905-f3306a1d16b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:10:10.709371 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.709337 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "901e16e6-76b0-426c-8905-f3306a1d16b0" (UID: "901e16e6-76b0-426c-8905-f3306a1d16b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:10:10.709505 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.709373 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901e16e6-76b0-426c-8905-f3306a1d16b0-kube-api-access-824fv" (OuterVolumeSpecName: "kube-api-access-824fv") pod "901e16e6-76b0-426c-8905-f3306a1d16b0" (UID: "901e16e6-76b0-426c-8905-f3306a1d16b0"). InnerVolumeSpecName "kube-api-access-824fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:10:10.807884 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.807844 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/901e16e6-76b0-426c-8905-f3306a1d16b0-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:10.807884 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.807877 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824fv\" (UniqueName: \"kubernetes.io/projected/901e16e6-76b0-426c-8905-f3306a1d16b0-kube-api-access-824fv\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:10.807884 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:10.807887 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/901e16e6-76b0-426c-8905-f3306a1d16b0-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:11.572059 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:11.572004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" event={"ID":"901e16e6-76b0-426c-8905-f3306a1d16b0","Type":"ContainerDied","Data":"b4f27f62131c3c2752c9a55a4edef4c408ab1950221ff63da90463aeb73d6f22"} Jul 22 13:10:11.572542 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:11.572078 2576 scope.go:117] "RemoveContainer" containerID="052e66024c8942d4aa51cb2e12908fde466a276dab3bff9290569bf3324f9d6f" Jul 22 13:10:11.572542 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:11.572034 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh" Jul 22 13:10:11.591296 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:11.591266 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh"] Jul 22 13:10:11.595519 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:11.595492 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0df68f3-b569b4f79-8x6sh"] Jul 22 13:10:12.713035 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:12.712984 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="901e16e6-76b0-426c-8905-f3306a1d16b0" path="/var/lib/kubelet/pods/901e16e6-76b0-426c-8905-f3306a1d16b0/volumes" Jul 22 13:10:13.199032 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:13.198984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:10:16.153589 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:16.153554 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph"] Jul 22 13:10:16.153961 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:16.153833 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" containerID="cri-o://236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7" gracePeriod=30 Jul 22 13:10:17.029944 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.029912 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt"] Jul 22 13:10:17.030318 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.030300 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="901e16e6-76b0-426c-8905-f3306a1d16b0" containerName="maas-api" Jul 22 13:10:17.030368 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.030322 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="901e16e6-76b0-426c-8905-f3306a1d16b0" containerName="maas-api" Jul 22 13:10:17.030409 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.030399 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="901e16e6-76b0-426c-8905-f3306a1d16b0" containerName="maas-api" Jul 22 13:10:17.033356 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.033339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.037639 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:10:17.037600 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" is forbidden: User \"system:node:ip-10-0-137-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-88.ec2.internal' and this object" podUID="25856a3b-207f-4830-a364-bc65a093c445" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.037766 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:10:17.037653 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-derive-8399da33-openshift-default-dockercfg-qfmd2\" is forbidden: User \"system:node:ip-10-0-137-88.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-88.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-derive-8399da33-openshift-default-dockercfg-qfmd2\"" type="*v1.Secret" Jul 22 13:10:17.059118 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.059093 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt"] Jul 22 13:10:17.165145 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165109 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25856a3b-207f-4830-a364-bc65a093c445-istiod-ca-cert\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.165518 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-credential-socket\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.165518 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2sx\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-kube-api-access-pv2sx\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.165518 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-istio-token\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.165518 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-data\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.165518 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-certs\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.165518 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25856a3b-207f-4830-a364-bc65a093c445-istio-podinfo\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.165518 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-envoy\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.165518 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.165459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-socket\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266636 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-credential-socket\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266645 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2sx\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-kube-api-access-pv2sx\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-istio-token\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266685 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-data\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-certs\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25856a3b-207f-4830-a364-bc65a093c445-istio-podinfo\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-envoy\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266796 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-socket\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.266825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.266821 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25856a3b-207f-4830-a364-bc65a093c445-istiod-ca-cert\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.267271 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.267198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-credential-socket\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.267271 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.267218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-data\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.267384 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.267271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-socket\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.267384 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.267350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-certs\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.267630 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.267605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25856a3b-207f-4830-a364-bc65a093c445-istiod-ca-cert\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.269335 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.269311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-envoy\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.269654 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.269630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25856a3b-207f-4830-a364-bc65a093c445-istio-podinfo\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.278719 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.278694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2sx\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-kube-api-access-pv2sx\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.282833 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.282786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-istio-token\") pod \"e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.840895 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.840861 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8399da33-openshift-default-dockercfg-qfmd2\"" Jul 22 13:10:17.845114 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.845091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:17.995338 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.995307 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt"] Jul 22 13:10:17.996404 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:10:17.996382 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25856a3b_207f_4830_a364_bc65a093c445.slice/crio-8b30ed3cb9654384577acf93c5b4273bac4ad68ba94eb2b7cfaf1e199e4e001d WatchSource:0}: Error finding container 8b30ed3cb9654384577acf93c5b4273bac4ad68ba94eb2b7cfaf1e199e4e001d: Status 404 returned error can't find the container with id 8b30ed3cb9654384577acf93c5b4273bac4ad68ba94eb2b7cfaf1e199e4e001d Jul 22 13:10:17.998740 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.998699 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:10:17.998828 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.998792 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:10:17.998868 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:17.998833 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:10:18.613093 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:18.613046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" event={"ID":"25856a3b-207f-4830-a364-bc65a093c445","Type":"ContainerStarted","Data":"e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5"} Jul 22 13:10:18.613610 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:18.613110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" event={"ID":"25856a3b-207f-4830-a364-bc65a093c445","Type":"ContainerStarted","Data":"8b30ed3cb9654384577acf93c5b4273bac4ad68ba94eb2b7cfaf1e199e4e001d"} Jul 22 13:10:18.641165 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:18.641086 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" podStartSLOduration=2.641068775 podStartE2EDuration="2.641068775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:10:18.640378286 +0000 UTC m=+1772.445095205" watchObservedRunningTime="2026-07-22 13:10:18.641068775 +0000 UTC m=+1772.445785671" Jul 22 13:10:18.845887 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:18.845851 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:19.850792 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:19.850765 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:20.121910 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.121830 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf"] Jul 22 13:10:20.125401 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.125386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.127621 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.127600 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8399da33-serving-cert\"" Jul 22 13:10:20.136193 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.136170 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf"] Jul 22 13:10:20.194286 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.194260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-openshift-service-ca\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.194436 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.194329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8pm6\" (UniqueName: \"kubernetes.io/projected/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-kube-api-access-g8pm6\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.194436 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.194398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.295922 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.295886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-openshift-service-ca\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.296146 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.295981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8pm6\" (UniqueName: \"kubernetes.io/projected/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-kube-api-access-g8pm6\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.296146 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.296053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.296235 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:10:20.296199 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-serving-cert: secret "maas-api-e2e-derive-8399da33-serving-cert" not found Jul 22 13:10:20.296275 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:10:20.296267 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls podName:cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43 nodeName:}" failed. No retries permitted until 2026-07-22 13:10:20.796250026 +0000 UTC m=+1774.600966904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls") pod "maas-api-e2e-derive-8399da33-59f485b756-2bgbf" (UID: "cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43") : secret "maas-api-e2e-derive-8399da33-serving-cert" not found Jul 22 13:10:20.296521 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.296502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-openshift-service-ca\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.305803 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.305783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8pm6\" (UniqueName: \"kubernetes.io/projected/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-kube-api-access-g8pm6\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.623829 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.623797 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:20.624757 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.624739 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:20.800806 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.800762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:20.803533 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:20.803506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls\") pod \"maas-api-e2e-derive-8399da33-59f485b756-2bgbf\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:21.037565 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.037533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:21.188816 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.188789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf"] Jul 22 13:10:21.215949 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:10:21.215919 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3fa3d4_7aaa_4370_a7c8_7a482d9b2c43.slice/crio-7218d64e93327c8b27adcf05a7536ab7cea6e0bd87e5548f87f4f9137f80d201 WatchSource:0}: Error finding container 7218d64e93327c8b27adcf05a7536ab7cea6e0bd87e5548f87f4f9137f80d201: Status 404 returned error can't find the container with id 7218d64e93327c8b27adcf05a7536ab7cea6e0bd87e5548f87f4f9137f80d201 Jul 22 13:10:21.411853 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.411827 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:10:21.507946 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.507920 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637e437-eb09-4258-95dc-6e1ec75b879f-istiod-ca-cert\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508134 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.507988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-socket\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508134 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508071 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqgc2\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-kube-api-access-jqgc2\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508134 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508099 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-token\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508134 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508130 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-credential-socket\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508342 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-envoy\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508342 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508209 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-certs\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508342 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508246 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-data\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508342 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508286 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-podinfo\") pod \"8637e437-eb09-4258-95dc-6e1ec75b879f\" (UID: \"8637e437-eb09-4258-95dc-6e1ec75b879f\") " Jul 22 13:10:21.508342 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508296 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:10:21.508342 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508321 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8637e437-eb09-4258-95dc-6e1ec75b879f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:10:21.508692 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508426 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:10:21.508692 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508540 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:10:21.508692 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508644 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.508692 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508660 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-credential-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.508692 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508676 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-workload-certs\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.508692 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.508690 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8637e437-eb09-4258-95dc-6e1ec75b879f-istiod-ca-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.509074 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.509052 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:10:21.510523 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.510499 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-kube-api-access-jqgc2" (OuterVolumeSpecName: "kube-api-access-jqgc2") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "kube-api-access-jqgc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:10:21.510700 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.510677 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:10:21.510994 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.510968 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:10:21.511100 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.510971 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8637e437-eb09-4258-95dc-6e1ec75b879f" (UID: "8637e437-eb09-4258-95dc-6e1ec75b879f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:10:21.609657 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.609583 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqgc2\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-kube-api-access-jqgc2\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.609657 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.609608 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.609657 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.609617 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-envoy\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.609657 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.609625 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-data\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.609657 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.609632 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8637e437-eb09-4258-95dc-6e1ec75b879f-istio-podinfo\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:21.631232 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.631199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" event={"ID":"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43","Type":"ContainerStarted","Data":"7218d64e93327c8b27adcf05a7536ab7cea6e0bd87e5548f87f4f9137f80d201"} Jul 22 13:10:21.632845 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.632822 2576 generic.go:356] "Generic (PLEG): container finished" podID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerID="236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7" exitCode=0 Jul 22 13:10:21.632941 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.632879 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" Jul 22 13:10:21.632941 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.632913 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" event={"ID":"8637e437-eb09-4258-95dc-6e1ec75b879f","Type":"ContainerDied","Data":"236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7"} Jul 22 13:10:21.633041 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.632968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph" event={"ID":"8637e437-eb09-4258-95dc-6e1ec75b879f","Type":"ContainerDied","Data":"78f30aff463c4847d812f66809ecbc70fd746911b0857007b6c8244fd64d5321"} Jul 22 13:10:21.633041 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.632998 2576 scope.go:117] "RemoveContainer" containerID="236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7" Jul 22 13:10:21.643343 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.643325 2576 scope.go:117] "RemoveContainer" containerID="f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941" Jul 22 13:10:21.651667 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.651646 2576 scope.go:117] "RemoveContainer" containerID="236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7" Jul 22 13:10:21.651909 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:10:21.651891 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7\": container with ID starting with 236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7 not found: ID does not exist" containerID="236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7" Jul 22 13:10:21.651989 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.651923 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7"} err="failed to get container status \"236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7\": rpc error: code = NotFound desc = could not find container \"236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7\": container with ID starting with 236104d53d63bbd9d3fce6dd9f1f990e82b3172ca72bdb3354d42839737a1be7 not found: ID does not exist" Jul 22 13:10:21.651989 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.651947 2576 scope.go:117] "RemoveContainer" containerID="f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941" Jul 22 13:10:21.652209 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:10:21.652192 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941\": container with ID starting with f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941 not found: ID does not exist" containerID="f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941" Jul 22 13:10:21.652262 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.652218 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941"} err="failed to get container status \"f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941\": rpc error: code = NotFound desc = could not find container \"f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941\": container with ID starting with f71ecf9aeab5a958138eabfa83581456568f318851e687fa1b8e1acbbb239941 not found: ID does not exist" Jul 22 13:10:21.657673 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.657653 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph"] Jul 22 13:10:21.665236 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:21.665215 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e0df68f3-openshift-default-85bc7f464c-df8ph"] Jul 22 13:10:22.713372 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:22.713329 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" path="/var/lib/kubelet/pods/8637e437-eb09-4258-95dc-6e1ec75b879f/volumes" Jul 22 13:10:23.649409 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:23.649369 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" event={"ID":"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43","Type":"ContainerStarted","Data":"3506f41ff584fd4496759718b54f124695b33535db4f92248301dbc5f5d2e862"} Jul 22 13:10:23.649636 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:23.649475 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:23.671492 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:23.671454 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" podStartSLOduration=1.442207874 podStartE2EDuration="3.671441237s" totalImagesPullingTime="2.229233363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:10:23.669712923 +0000 UTC m=+1777.474429820" watchObservedRunningTime="2026-07-22 13:10:23.671441237 +0000 UTC m=+1777.476158112" Jul 22 13:10:27.500562 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:27.500528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:10:29.661759 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:29.661734 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:35.626177 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:35.626088 2576 patch_prober.go:28] interesting pod/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 22 13:10:35.626177 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:35.626155 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 22 13:10:35.720616 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:35.720585 2576 generic.go:356] "Generic (PLEG): container finished" podID="25856a3b-207f-4830-a364-bc65a093c445" containerID="e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5" exitCode=0 Jul 22 13:10:35.720777 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:35.720652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" event={"ID":"25856a3b-207f-4830-a364-bc65a093c445","Type":"ContainerDied","Data":"e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5"} Jul 22 13:10:35.721124 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:35.721106 2576 scope.go:117] "RemoveContainer" containerID="e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5" Jul 22 13:10:35.722150 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:35.722120 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:10:35.722225 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:35.722207 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:10:35.722268 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:35.722253 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:10:36.727147 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:36.727112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" event={"ID":"25856a3b-207f-4830-a364-bc65a093c445","Type":"ContainerStarted","Data":"e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d"} Jul 22 13:10:36.846132 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:36.846094 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:36.846401 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:36.846378 2576 patch_prober.go:28] interesting pod/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 22 13:10:36.846499 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:36.846445 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 22 13:10:37.845740 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:37.845706 2576 patch_prober.go:28] interesting pod/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 22 13:10:37.846163 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:37.845761 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 22 13:10:38.845646 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:38.845611 2576 patch_prober.go:28] interesting pod/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 22 13:10:38.845816 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:38.845675 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 22 13:10:39.846339 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:39.846303 2576 patch_prober.go:28] interesting pod/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 22 13:10:39.846744 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:39.846356 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 22 13:10:40.845646 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:40.845611 2576 patch_prober.go:28] interesting pod/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 22 13:10:40.845829 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:40.845669 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 22 13:10:41.850088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:41.850057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:41.850519 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:41.850475 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:41.851196 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:41.851179 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:10:46.748779 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:46.748754 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:10:46.757783 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:46.757766 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:10:46.758920 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:46.758902 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:10:46.765240 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:46.765221 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:10:55.579292 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.579251 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf"] Jul 22 13:10:55.579836 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.579492 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" podUID="cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" containerName="maas-api" containerID="cri-o://3506f41ff584fd4496759718b54f124695b33535db4f92248301dbc5f5d2e862" gracePeriod=30 Jul 22 13:10:55.831347 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.831278 2576 generic.go:356] "Generic (PLEG): container finished" podID="cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" containerID="3506f41ff584fd4496759718b54f124695b33535db4f92248301dbc5f5d2e862" exitCode=0 Jul 22 13:10:55.831470 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.831352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" event={"ID":"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43","Type":"ContainerDied","Data":"3506f41ff584fd4496759718b54f124695b33535db4f92248301dbc5f5d2e862"} Jul 22 13:10:55.846519 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.846497 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:55.911349 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.911324 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls\") pod \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " Jul 22 13:10:55.911469 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.911361 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8pm6\" (UniqueName: \"kubernetes.io/projected/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-kube-api-access-g8pm6\") pod \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " Jul 22 13:10:55.911469 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.911429 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-openshift-service-ca\") pod \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\" (UID: \"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43\") " Jul 22 13:10:55.911879 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.911855 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" (UID: "cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:10:55.913702 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.913674 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" (UID: "cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:10:55.913801 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:55.913745 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-kube-api-access-g8pm6" (OuterVolumeSpecName: "kube-api-access-g8pm6") pod "cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" (UID: "cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43"). InnerVolumeSpecName "kube-api-access-g8pm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:10:56.012190 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:56.012148 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:56.012190 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:56.012176 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:56.012190 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:56.012185 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8pm6\" (UniqueName: \"kubernetes.io/projected/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43-kube-api-access-g8pm6\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:10:56.837325 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:56.837292 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" Jul 22 13:10:56.837730 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:56.837322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf" event={"ID":"cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43","Type":"ContainerDied","Data":"7218d64e93327c8b27adcf05a7536ab7cea6e0bd87e5548f87f4f9137f80d201"} Jul 22 13:10:56.837730 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:56.837383 2576 scope.go:117] "RemoveContainer" containerID="3506f41ff584fd4496759718b54f124695b33535db4f92248301dbc5f5d2e862" Jul 22 13:10:56.856443 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:56.856419 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf"] Jul 22 13:10:56.860626 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:56.860604 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8399da33-59f485b756-2bgbf"] Jul 22 13:10:58.403965 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:58.403932 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:10:58.713289 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:10:58.713255 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" path="/var/lib/kubelet/pods/cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43/volumes" Jul 22 13:11:00.225611 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:00.225577 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt"] Jul 22 13:11:00.226001 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:00.225863 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" containerID="cri-o://e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d" gracePeriod=30 Jul 22 13:11:05.485379 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.485356 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:11:05.590468 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590389 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-envoy\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.590468 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590433 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-certs\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.590703 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590473 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv2sx\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-kube-api-access-pv2sx\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.590703 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590536 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-istio-token\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.590703 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590561 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-credential-socket\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.590703 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590589 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25856a3b-207f-4830-a364-bc65a093c445-istio-podinfo\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.590703 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590622 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-socket\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.590703 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-data\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.591045 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590713 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25856a3b-207f-4830-a364-bc65a093c445-istiod-ca-cert\") pod \"25856a3b-207f-4830-a364-bc65a093c445\" (UID: \"25856a3b-207f-4830-a364-bc65a093c445\") " Jul 22 13:11:05.591045 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590841 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:11:05.591045 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.590903 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:11:05.591218 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.591095 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-certs\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.591218 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.591116 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-workload-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.591218 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.591168 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:11:05.591392 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.591281 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25856a3b-207f-4830-a364-bc65a093c445-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:11:05.592240 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.592211 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-data" (OuterVolumeSpecName: "istio-data") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:11:05.592895 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.592871 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-kube-api-access-pv2sx" (OuterVolumeSpecName: "kube-api-access-pv2sx") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "kube-api-access-pv2sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:11:05.592977 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.592899 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:11:05.593283 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.593242 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25856a3b-207f-4830-a364-bc65a093c445-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:11:05.593283 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.593256 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-istio-token" (OuterVolumeSpecName: "istio-token") pod "25856a3b-207f-4830-a364-bc65a093c445" (UID: "25856a3b-207f-4830-a364-bc65a093c445"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:11:05.691991 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.691955 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-istio-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.691991 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.691984 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-credential-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.691991 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.691994 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25856a3b-207f-4830-a364-bc65a093c445-istio-podinfo\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.691991 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.692002 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-data\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.692271 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.692031 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25856a3b-207f-4830-a364-bc65a093c445-istiod-ca-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.692271 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.692040 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25856a3b-207f-4830-a364-bc65a093c445-istio-envoy\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.692271 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.692047 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv2sx\" (UniqueName: \"kubernetes.io/projected/25856a3b-207f-4830-a364-bc65a093c445-kube-api-access-pv2sx\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:11:05.891306 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.891220 2576 generic.go:356] "Generic (PLEG): container finished" podID="25856a3b-207f-4830-a364-bc65a093c445" containerID="e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d" exitCode=0 Jul 22 13:11:05.891306 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.891277 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" Jul 22 13:11:05.891306 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.891291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" event={"ID":"25856a3b-207f-4830-a364-bc65a093c445","Type":"ContainerDied","Data":"e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d"} Jul 22 13:11:05.891532 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.891333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt" event={"ID":"25856a3b-207f-4830-a364-bc65a093c445","Type":"ContainerDied","Data":"8b30ed3cb9654384577acf93c5b4273bac4ad68ba94eb2b7cfaf1e199e4e001d"} Jul 22 13:11:05.891532 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.891355 2576 scope.go:117] "RemoveContainer" containerID="e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d" Jul 22 13:11:05.901497 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.901481 2576 scope.go:117] "RemoveContainer" containerID="e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5" Jul 22 13:11:05.909982 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.909963 2576 scope.go:117] "RemoveContainer" containerID="e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d" Jul 22 13:11:05.910250 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:11:05.910229 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d\": container with ID starting with e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d not found: ID does not exist" containerID="e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d" Jul 22 13:11:05.910302 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.910259 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d"} err="failed to get container status \"e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d\": rpc error: code = NotFound desc = could not find container \"e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d\": container with ID starting with e8d2f46af7fa57ce2fd28b2f4924ed94742b870b3ae35784a44fc157cc418c9d not found: ID does not exist" Jul 22 13:11:05.910302 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.910277 2576 scope.go:117] "RemoveContainer" containerID="e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5" Jul 22 13:11:05.910475 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:11:05.910455 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5\": container with ID starting with e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5 not found: ID does not exist" containerID="e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5" Jul 22 13:11:05.910525 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.910482 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5"} err="failed to get container status \"e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5\": rpc error: code = NotFound desc = could not find container \"e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5\": container with ID starting with e430d17565b3d5713e9cf18da969afa848052f4ec9f256a7d152ab32b0044cf5 not found: ID does not exist" Jul 22 13:11:05.915390 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.915368 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt"] Jul 22 13:11:05.919791 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:05.919770 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8399da33-openshift-default-86f5b69679-9n2mt"] Jul 22 13:11:06.713151 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:11:06.713114 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25856a3b-207f-4830-a364-bc65a093c445" path="/var/lib/kubelet/pods/25856a3b-207f-4830-a364-bc65a093c445/volumes" Jul 22 13:12:28.822500 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:28.822426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:12:40.798347 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:40.798307 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:12:42.989127 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989087 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw"] Jul 22 13:12:42.989501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989450 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" Jul 22 13:12:42.989501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989461 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" Jul 22 13:12:42.989501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989472 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" Jul 22 13:12:42.989501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989477 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" Jul 22 13:12:42.989501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989486 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" Jul 22 13:12:42.989501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989491 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" Jul 22 13:12:42.989501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989498 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" Jul 22 13:12:42.989501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989504 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" Jul 22 13:12:42.989850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989512 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" containerName="maas-api" Jul 22 13:12:42.989850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989517 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" containerName="maas-api" Jul 22 13:12:42.989850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989572 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3fa3d4-7aaa-4370-a7c8-7a482d9b2c43" containerName="maas-api" Jul 22 13:12:42.989850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989586 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" Jul 22 13:12:42.989850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989592 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="25856a3b-207f-4830-a364-bc65a093c445" containerName="istio-proxy" Jul 22 13:12:42.989850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989600 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" Jul 22 13:12:42.989850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.989606 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8637e437-eb09-4258-95dc-6e1ec75b879f" containerName="istio-proxy" Jul 22 13:12:42.992278 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.992260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:42.995683 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:42.995666 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-79905e7d-openshift-default-dockercfg-zxt8h\"" Jul 22 13:12:43.016585 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-certs\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.016691 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-envoy\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.016691 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-socket\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.016830 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfcqh\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-kube-api-access-gfcqh\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.016900 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-token\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.016900 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-podinfo\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.017006 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-credential-socket\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.017006 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istiod-ca-cert\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.017006 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.016995 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-data\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.019952 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.019928 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw"] Jul 22 13:12:43.117883 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.117842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-data\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.117893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-certs\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.117922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-envoy\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.117972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-socket\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118033 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfcqh\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-kube-api-access-gfcqh\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-token\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-podinfo\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118373 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-credential-socket\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118373 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118154 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istiod-ca-cert\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118373 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118363 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-data\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118699 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118552 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-socket\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118699 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-credential-socket\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118813 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-certs\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.118874 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.118855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istiod-ca-cert\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.120483 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.120463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-envoy\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.120789 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.120769 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-podinfo\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.126655 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.126629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfcqh\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-kube-api-access-gfcqh\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.127067 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.127047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-token\") pod \"e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.302754 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.302296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:43.446662 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.446634 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw"] Jul 22 13:12:43.448191 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:12:43.448165 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ec1586_6f6e_4541_96d9_2f6145d3d62d.slice/crio-4f19f4b81376ad4ddb5bdf77abd652a73d15466b5cb4b65fcd5a16b580f0f73c WatchSource:0}: Error finding container 4f19f4b81376ad4ddb5bdf77abd652a73d15466b5cb4b65fcd5a16b580f0f73c: Status 404 returned error can't find the container with id 4f19f4b81376ad4ddb5bdf77abd652a73d15466b5cb4b65fcd5a16b580f0f73c Jul 22 13:12:43.450186 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.450145 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:12:43.450276 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.450218 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:12:43.450276 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:43.450260 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:12:44.418908 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:44.418872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" event={"ID":"f2ec1586-6f6e-4541-96d9-2f6145d3d62d","Type":"ContainerStarted","Data":"9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f"} Jul 22 13:12:44.419372 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:44.418954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" event={"ID":"f2ec1586-6f6e-4541-96d9-2f6145d3d62d","Type":"ContainerStarted","Data":"4f19f4b81376ad4ddb5bdf77abd652a73d15466b5cb4b65fcd5a16b580f0f73c"} Jul 22 13:12:44.441229 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:44.441179 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podStartSLOduration=2.441165403 podStartE2EDuration="2.441165403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:12:44.439370754 +0000 UTC m=+1918.244087651" watchObservedRunningTime="2026-07-22 13:12:44.441165403 +0000 UTC m=+1918.245882299" Jul 22 13:12:45.303043 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:45.302977 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:45.308739 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:45.308712 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:45.423984 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:45.423956 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:45.424839 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:45.424821 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:12:46.635074 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.635041 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27"] Jul 22 13:12:46.638812 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.638796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.641212 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.641194 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-79905e7d-serving-cert\"" Jul 22 13:12:46.647146 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.646909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfdcd6dc-4632-40ef-b96a-d262e86aca79-maas-api-tls\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.647146 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.647068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdcd6dc-4632-40ef-b96a-d262e86aca79-openshift-service-ca\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.647286 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.647205 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nldfk\" (UniqueName: \"kubernetes.io/projected/cfdcd6dc-4632-40ef-b96a-d262e86aca79-kube-api-access-nldfk\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.651187 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.651165 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27"] Jul 22 13:12:46.748501 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.748471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nldfk\" (UniqueName: \"kubernetes.io/projected/cfdcd6dc-4632-40ef-b96a-d262e86aca79-kube-api-access-nldfk\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.748650 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.748538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfdcd6dc-4632-40ef-b96a-d262e86aca79-maas-api-tls\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.748650 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.748616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdcd6dc-4632-40ef-b96a-d262e86aca79-openshift-service-ca\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.749298 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.749269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdcd6dc-4632-40ef-b96a-d262e86aca79-openshift-service-ca\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.751087 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.751067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfdcd6dc-4632-40ef-b96a-d262e86aca79-maas-api-tls\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.759844 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.759823 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nldfk\" (UniqueName: \"kubernetes.io/projected/cfdcd6dc-4632-40ef-b96a-d262e86aca79-kube-api-access-nldfk\") pod \"maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:46.953260 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:46.953236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:47.099865 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:47.099836 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27"] Jul 22 13:12:47.100387 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:12:47.100360 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdcd6dc_4632_40ef_b96a_d262e86aca79.slice/crio-bec2fe11936d0449e480005aa23085748c3d38819485fd6a91fc7b3624468c3a WatchSource:0}: Error finding container bec2fe11936d0449e480005aa23085748c3d38819485fd6a91fc7b3624468c3a: Status 404 returned error can't find the container with id bec2fe11936d0449e480005aa23085748c3d38819485fd6a91fc7b3624468c3a Jul 22 13:12:47.437109 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:47.437074 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" event={"ID":"cfdcd6dc-4632-40ef-b96a-d262e86aca79","Type":"ContainerStarted","Data":"bec2fe11936d0449e480005aa23085748c3d38819485fd6a91fc7b3624468c3a"} Jul 22 13:12:50.455350 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:50.455315 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" event={"ID":"cfdcd6dc-4632-40ef-b96a-d262e86aca79","Type":"ContainerStarted","Data":"824fda179cbdcfcfc37afe1488f4f4564cf5241c6d9f791d05d93b2195eacc7b"} Jul 22 13:12:50.455719 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:50.455397 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:12:50.476828 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:50.476765 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" podStartSLOduration=1.572447741 podStartE2EDuration="4.476750775s" totalImagesPullingTime="2.904303034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:12:50.474232216 +0000 UTC m=+1924.278949134" watchObservedRunningTime="2026-07-22 13:12:50.476750775 +0000 UTC m=+1924.281467670" Jul 22 13:12:53.467883 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:53.467843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:12:56.467687 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:12:56.467653 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:13:00.771666 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:00.771630 2576 patch_prober.go:28] interesting pod/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:45312->10.132.0.57:15021: read: connection reset by peer" start-of-body= Jul 22 13:13:00.772093 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:00.771704 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:45312->10.132.0.57:15021: read: connection reset by peer" Jul 22 13:13:01.528606 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:01.528570 2576 generic.go:356] "Generic (PLEG): container finished" podID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerID="9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f" exitCode=0 Jul 22 13:13:01.528773 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:01.528642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" event={"ID":"f2ec1586-6f6e-4541-96d9-2f6145d3d62d","Type":"ContainerDied","Data":"9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f"} Jul 22 13:13:01.529040 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:01.529005 2576 scope.go:117] "RemoveContainer" containerID="9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f" Jul 22 13:13:01.530088 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:01.530058 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:13:01.530188 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:01.530124 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:13:01.530188 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:01.530153 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:13:02.535994 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:02.535954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" event={"ID":"f2ec1586-6f6e-4541-96d9-2f6145d3d62d","Type":"ContainerStarted","Data":"056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9"} Jul 22 13:13:03.303390 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:03.303358 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:13:03.303556 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:03.303453 2576 patch_prober.go:28] interesting pod/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 22 13:13:03.303556 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:03.303497 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 22 13:13:04.303217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:04.303189 2576 patch_prober.go:28] interesting pod/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 22 13:13:04.303604 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:04.303251 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 22 13:13:05.304393 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:05.303914 2576 patch_prober.go:28] interesting pod/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 22 13:13:05.304393 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:05.303996 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 22 13:13:06.303705 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:06.303671 2576 patch_prober.go:28] interesting pod/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 22 13:13:06.304006 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:06.303976 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 22 13:13:07.303722 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:07.303680 2576 patch_prober.go:28] interesting pod/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 22 13:13:07.304224 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:07.303745 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 22 13:13:08.303819 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:08.303774 2576 patch_prober.go:28] interesting pod/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 22 13:13:08.304341 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:08.303840 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 22 13:13:09.306603 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.306574 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:13:09.307039 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.306913 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:13:09.307919 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.307893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:13:09.328752 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.328726 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27"] Jul 22 13:13:09.329465 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.329057 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" podUID="cfdcd6dc-4632-40ef-b96a-d262e86aca79" containerName="maas-api" containerID="cri-o://824fda179cbdcfcfc37afe1488f4f4564cf5241c6d9f791d05d93b2195eacc7b" gracePeriod=30 Jul 22 13:13:09.577939 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.577913 2576 generic.go:356] "Generic (PLEG): container finished" podID="cfdcd6dc-4632-40ef-b96a-d262e86aca79" containerID="824fda179cbdcfcfc37afe1488f4f4564cf5241c6d9f791d05d93b2195eacc7b" exitCode=0 Jul 22 13:13:09.578105 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.577981 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" event={"ID":"cfdcd6dc-4632-40ef-b96a-d262e86aca79","Type":"ContainerDied","Data":"824fda179cbdcfcfc37afe1488f4f4564cf5241c6d9f791d05d93b2195eacc7b"} Jul 22 13:13:09.578105 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.578051 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" event={"ID":"cfdcd6dc-4632-40ef-b96a-d262e86aca79","Type":"ContainerDied","Data":"bec2fe11936d0449e480005aa23085748c3d38819485fd6a91fc7b3624468c3a"} Jul 22 13:13:09.578105 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.578070 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bec2fe11936d0449e480005aa23085748c3d38819485fd6a91fc7b3624468c3a" Jul 22 13:13:09.585311 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.585295 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:13:09.655319 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.655289 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfdcd6dc-4632-40ef-b96a-d262e86aca79-maas-api-tls\") pod \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " Jul 22 13:13:09.655496 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.655326 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nldfk\" (UniqueName: \"kubernetes.io/projected/cfdcd6dc-4632-40ef-b96a-d262e86aca79-kube-api-access-nldfk\") pod \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " Jul 22 13:13:09.655496 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.655453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdcd6dc-4632-40ef-b96a-d262e86aca79-openshift-service-ca\") pod \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\" (UID: \"cfdcd6dc-4632-40ef-b96a-d262e86aca79\") " Jul 22 13:13:09.657774 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.655760 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfdcd6dc-4632-40ef-b96a-d262e86aca79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfdcd6dc-4632-40ef-b96a-d262e86aca79" (UID: "cfdcd6dc-4632-40ef-b96a-d262e86aca79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:13:09.658376 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.658346 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdcd6dc-4632-40ef-b96a-d262e86aca79-kube-api-access-nldfk" (OuterVolumeSpecName: "kube-api-access-nldfk") pod "cfdcd6dc-4632-40ef-b96a-d262e86aca79" (UID: "cfdcd6dc-4632-40ef-b96a-d262e86aca79"). InnerVolumeSpecName "kube-api-access-nldfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:13:09.658517 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.658397 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfdcd6dc-4632-40ef-b96a-d262e86aca79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfdcd6dc-4632-40ef-b96a-d262e86aca79" (UID: "cfdcd6dc-4632-40ef-b96a-d262e86aca79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:13:09.757037 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.756984 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfdcd6dc-4632-40ef-b96a-d262e86aca79-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:09.757037 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.757037 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfdcd6dc-4632-40ef-b96a-d262e86aca79-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:09.757179 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:09.757049 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nldfk\" (UniqueName: \"kubernetes.io/projected/cfdcd6dc-4632-40ef-b96a-d262e86aca79-kube-api-access-nldfk\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:10.583347 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:10.583317 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27" Jul 22 13:13:10.609383 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:10.609359 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27"] Jul 22 13:13:10.611339 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:10.611319 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79905e7d-c784c7bbd-rmr27"] Jul 22 13:13:10.713107 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:10.713083 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfdcd6dc-4632-40ef-b96a-d262e86aca79" path="/var/lib/kubelet/pods/cfdcd6dc-4632-40ef-b96a-d262e86aca79/volumes" Jul 22 13:13:12.002562 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:12.002528 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:13:20.797438 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:20.797408 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw"] Jul 22 13:13:20.797867 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:20.797630 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" containerID="cri-o://056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9" gracePeriod=30 Jul 22 13:13:24.309284 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:24.309233 2576 patch_prober.go:28] interesting pod/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 22 13:13:24.309721 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:24.309316 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 13:13:26.046652 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.046629 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:13:26.187466 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187426 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-socket\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187656 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187476 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-token\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187656 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187514 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-envoy\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187656 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187539 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-podinfo\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187656 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187570 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-credential-socket\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187656 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187613 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istiod-ca-cert\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187656 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187640 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfcqh\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-kube-api-access-gfcqh\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187988 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187699 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-certs\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187988 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187720 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-data\") pod \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\" (UID: \"f2ec1586-6f6e-4541-96d9-2f6145d3d62d\") " Jul 22 13:13:26.187988 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187724 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:13:26.187988 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.187876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:13:26.188215 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.188002 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.188215 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.188049 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-credential-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.188215 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.188043 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:13:26.188215 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.188212 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:13:26.188849 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.188823 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:13:26.189889 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.189862 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:13:26.190040 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.189932 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:13:26.190232 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.190213 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-kube-api-access-gfcqh" (OuterVolumeSpecName: "kube-api-access-gfcqh") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "kube-api-access-gfcqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:13:26.190327 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.190232 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2ec1586-6f6e-4541-96d9-2f6145d3d62d" (UID: "f2ec1586-6f6e-4541-96d9-2f6145d3d62d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:13:26.288410 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.288387 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-workload-certs\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.288410 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.288407 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-data\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.288547 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.288416 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.288547 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.288424 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-envoy\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.288547 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.288431 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istio-podinfo\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.288547 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.288438 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-istiod-ca-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.288547 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.288446 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfcqh\" (UniqueName: \"kubernetes.io/projected/f2ec1586-6f6e-4541-96d9-2f6145d3d62d-kube-api-access-gfcqh\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:13:26.672424 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.672390 2576 generic.go:356] "Generic (PLEG): container finished" podID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerID="056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9" exitCode=0 Jul 22 13:13:26.672567 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.672442 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" Jul 22 13:13:26.672567 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.672480 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" event={"ID":"f2ec1586-6f6e-4541-96d9-2f6145d3d62d","Type":"ContainerDied","Data":"056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9"} Jul 22 13:13:26.672567 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.672534 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw" event={"ID":"f2ec1586-6f6e-4541-96d9-2f6145d3d62d","Type":"ContainerDied","Data":"4f19f4b81376ad4ddb5bdf77abd652a73d15466b5cb4b65fcd5a16b580f0f73c"} Jul 22 13:13:26.672567 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.672560 2576 scope.go:117] "RemoveContainer" containerID="056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9" Jul 22 13:13:26.681849 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.681833 2576 scope.go:117] "RemoveContainer" containerID="9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f" Jul 22 13:13:26.690057 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.690039 2576 scope.go:117] "RemoveContainer" containerID="056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9" Jul 22 13:13:26.690285 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:13:26.690267 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9\": container with ID starting with 056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9 not found: ID does not exist" containerID="056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9" Jul 22 13:13:26.690327 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.690309 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9"} err="failed to get container status \"056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9\": rpc error: code = NotFound desc = could not find container \"056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9\": container with ID starting with 056829ea17c1fed49675ff3b3c1e43fcf4adc6f4c96712ab0a145fc25ddd29d9 not found: ID does not exist" Jul 22 13:13:26.690368 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.690325 2576 scope.go:117] "RemoveContainer" containerID="9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f" Jul 22 13:13:26.690527 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:13:26.690509 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f\": container with ID starting with 9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f not found: ID does not exist" containerID="9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f" Jul 22 13:13:26.690576 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.690531 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f"} err="failed to get container status \"9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f\": rpc error: code = NotFound desc = could not find container \"9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f\": container with ID starting with 9de50194754b938946d994fec16e775eb96adc39f97b0aeb191baed4ac82f12f not found: ID does not exist" Jul 22 13:13:26.696050 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.696029 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw"] Jul 22 13:13:26.700886 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.700866 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79905e7d-openshift-default-8f5d7b7c9-qr8nw"] Jul 22 13:13:26.712410 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:26.712391 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" path="/var/lib/kubelet/pods/f2ec1586-6f6e-4541-96d9-2f6145d3d62d/volumes" Jul 22 13:13:41.714953 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.714880 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst"] Jul 22 13:13:41.715377 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.715268 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" Jul 22 13:13:41.715377 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.715284 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" Jul 22 13:13:41.715377 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.715295 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" Jul 22 13:13:41.715377 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.715300 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" Jul 22 13:13:41.715377 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.715317 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfdcd6dc-4632-40ef-b96a-d262e86aca79" containerName="maas-api" Jul 22 13:13:41.715377 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.715322 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfdcd6dc-4632-40ef-b96a-d262e86aca79" containerName="maas-api" Jul 22 13:13:41.715561 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.715383 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" Jul 22 13:13:41.715561 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.715391 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfdcd6dc-4632-40ef-b96a-d262e86aca79" containerName="maas-api" Jul 22 13:13:41.717988 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.717972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.720680 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.720658 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7caca13d-openshift-default-dockercfg-gplsc\"" Jul 22 13:13:41.732213 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.732188 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst"] Jul 22 13:13:41.809750 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.809706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-data\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.809937 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.809762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-socket\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.809937 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.809792 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-podinfo\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.809937 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.809851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-certs\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.809937 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.809927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-credential-socket\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.810210 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.809982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istiod-ca-cert\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.810210 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.810045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-token\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.810210 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.810088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-envoy\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.810210 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.810115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrl4\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-kube-api-access-snrl4\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.910800 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.910769 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istiod-ca-cert\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.910800 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.910804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-token\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911055 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.910825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-envoy\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911055 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.910840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrl4\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-kube-api-access-snrl4\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911055 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.910896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-data\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911055 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.910915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-socket\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911055 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.910937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-podinfo\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911055 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.910987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-certs\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911055 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.911040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-credential-socket\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911452 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.911425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-certs\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911552 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.911446 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-data\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911552 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.911436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-credential-socket\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911552 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.911507 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-socket\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.911778 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.911756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istiod-ca-cert\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.913286 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.913248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-envoy\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.913863 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.913844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-podinfo\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.920097 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.920073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-token\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:41.920332 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:41.920313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrl4\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-kube-api-access-snrl4\") pod \"e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:42.028456 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:42.028400 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:42.168859 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:42.168827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst"] Jul 22 13:13:42.170196 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:13:42.170172 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3d3b8f_ce7e_4721_a5b0_90c2ad63d10f.slice/crio-33df8cc9cc3cc56f09a8ae6802fdf15f4bf110d3cc4f0852c24e8d4885c207f7 WatchSource:0}: Error finding container 33df8cc9cc3cc56f09a8ae6802fdf15f4bf110d3cc4f0852c24e8d4885c207f7: Status 404 returned error can't find the container with id 33df8cc9cc3cc56f09a8ae6802fdf15f4bf110d3cc4f0852c24e8d4885c207f7 Jul 22 13:13:42.172154 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:42.172123 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:13:42.172233 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:42.172191 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:13:42.172233 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:42.172222 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:13:42.757527 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:42.757493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" event={"ID":"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f","Type":"ContainerStarted","Data":"1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b"} Jul 22 13:13:42.757989 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:42.757547 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" event={"ID":"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f","Type":"ContainerStarted","Data":"33df8cc9cc3cc56f09a8ae6802fdf15f4bf110d3cc4f0852c24e8d4885c207f7"} Jul 22 13:13:42.813393 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:42.813335 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" podStartSLOduration=1.8133145590000002 podStartE2EDuration="1.813314559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:13:42.810906382 +0000 UTC m=+1976.615623278" watchObservedRunningTime="2026-07-22 13:13:42.813314559 +0000 UTC m=+1976.618031457" Jul 22 13:13:43.028995 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:43.028907 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:43.033742 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:43.033716 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:43.762946 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:43.762917 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:43.763852 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:43.763833 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:13:44.925264 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:44.925234 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8"] Jul 22 13:13:44.925829 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:44.925805 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ec1586-6f6e-4541-96d9-2f6145d3d62d" containerName="istio-proxy" Jul 22 13:13:44.928083 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:44.928063 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:44.930511 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:44.930492 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7caca13d-serving-cert\"" Jul 22 13:13:44.940251 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:44.940229 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8"] Jul 22 13:13:45.039173 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.039142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh9dv\" (UniqueName: \"kubernetes.io/projected/5365d238-b3fc-4ac8-8bca-495d2234bb3f-kube-api-access-wh9dv\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.039332 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.039206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.039332 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.039245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5365d238-b3fc-4ac8-8bca-495d2234bb3f-openshift-service-ca\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.140511 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.140486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.140644 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.140526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5365d238-b3fc-4ac8-8bca-495d2234bb3f-openshift-service-ca\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.140644 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.140631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh9dv\" (UniqueName: \"kubernetes.io/projected/5365d238-b3fc-4ac8-8bca-495d2234bb3f-kube-api-access-wh9dv\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.140716 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:13:45.140633 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-serving-cert: secret "maas-api-e2e-mt-7caca13d-serving-cert" not found Jul 22 13:13:45.140749 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:13:45.140741 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls podName:5365d238-b3fc-4ac8-8bca-495d2234bb3f nodeName:}" failed. No retries permitted until 2026-07-22 13:13:45.640725144 +0000 UTC m=+1979.445442021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls") pod "maas-api-e2e-mt-7caca13d-66794f487-jc5s8" (UID: "5365d238-b3fc-4ac8-8bca-495d2234bb3f") : secret "maas-api-e2e-mt-7caca13d-serving-cert" not found Jul 22 13:13:45.141090 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.141075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5365d238-b3fc-4ac8-8bca-495d2234bb3f-openshift-service-ca\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.150492 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.150468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh9dv\" (UniqueName: \"kubernetes.io/projected/5365d238-b3fc-4ac8-8bca-495d2234bb3f-kube-api-access-wh9dv\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.643686 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.643650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.646151 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.646132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls\") pod \"maas-api-e2e-mt-7caca13d-66794f487-jc5s8\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:45.839462 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:45.839433 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:46.182855 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:46.182828 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8"] Jul 22 13:13:46.184086 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:13:46.184056 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5365d238_b3fc_4ac8_8bca_495d2234bb3f.slice/crio-879a97eceed98dedb9d41b1551d1722b2dbe0f122e94269e5b588e9507dea866 WatchSource:0}: Error finding container 879a97eceed98dedb9d41b1551d1722b2dbe0f122e94269e5b588e9507dea866: Status 404 returned error can't find the container with id 879a97eceed98dedb9d41b1551d1722b2dbe0f122e94269e5b588e9507dea866 Jul 22 13:13:46.185537 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:46.185517 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 13:13:46.779651 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:46.779621 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" event={"ID":"5365d238-b3fc-4ac8-8bca-495d2234bb3f","Type":"ContainerStarted","Data":"f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5"} Jul 22 13:13:46.779798 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:46.779667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" event={"ID":"5365d238-b3fc-4ac8-8bca-495d2234bb3f","Type":"ContainerStarted","Data":"879a97eceed98dedb9d41b1551d1722b2dbe0f122e94269e5b588e9507dea866"} Jul 22 13:13:46.779798 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:46.779707 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:46.800713 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:46.800669 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" podStartSLOduration=2.345881282 podStartE2EDuration="2.800657007s" totalImagesPullingTime="454.775725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:13:46.798127938 +0000 UTC m=+1980.602844850" watchObservedRunningTime="2026-07-22 13:13:46.800657007 +0000 UTC m=+1980.605373960" Jul 22 13:13:51.796227 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:51.796193 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:13:52.789837 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:52.789808 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:13:59.301454 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:59.301411 2576 patch_prober.go:28] interesting pod/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:33782->10.132.0.59:15021: read: connection reset by peer" start-of-body= Jul 22 13:13:59.301868 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:59.301518 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:33782->10.132.0.59:15021: read: connection reset by peer" Jul 22 13:13:59.859174 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:59.859146 2576 generic.go:356] "Generic (PLEG): container finished" podID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerID="1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b" exitCode=0 Jul 22 13:13:59.859325 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:59.859220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" event={"ID":"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f","Type":"ContainerDied","Data":"1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b"} Jul 22 13:13:59.859637 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:59.859621 2576 scope.go:117] "RemoveContainer" containerID="1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b" Jul 22 13:13:59.860582 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:59.860546 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:13:59.860639 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:59.860617 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:13:59.860677 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:13:59.860647 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:14:00.867071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:00.867040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" event={"ID":"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f","Type":"ContainerStarted","Data":"719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9"} Jul 22 13:14:01.029119 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:01.029081 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:14:01.029444 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:01.029422 2576 patch_prober.go:28] interesting pod/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 22 13:14:01.029516 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:01.029471 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 22 13:14:02.029359 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:02.029326 2576 patch_prober.go:28] interesting pod/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 22 13:14:02.029719 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:02.029380 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 22 13:14:02.803525 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:02.803486 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:14:03.028999 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:03.028963 2576 patch_prober.go:28] interesting pod/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 22 13:14:03.029209 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:03.029053 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 22 13:14:04.029057 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:04.028995 2576 patch_prober.go:28] interesting pod/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 22 13:14:04.029429 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:04.029086 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 22 13:14:05.033079 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:05.033052 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:14:05.033438 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:05.033292 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:14:05.034096 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:05.034083 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:14:13.900754 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:13.900715 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:14:31.395608 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:31.395575 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:14:33.335936 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.335906 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8"] Jul 22 13:14:33.336364 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.336146 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" podUID="5365d238-b3fc-4ac8-8bca-495d2234bb3f" containerName="maas-api" containerID="cri-o://f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5" gracePeriod=30 Jul 22 13:14:33.591454 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.591394 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:14:33.754543 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.754509 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh9dv\" (UniqueName: \"kubernetes.io/projected/5365d238-b3fc-4ac8-8bca-495d2234bb3f-kube-api-access-wh9dv\") pod \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " Jul 22 13:14:33.754543 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.754549 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls\") pod \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " Jul 22 13:14:33.754748 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.754601 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5365d238-b3fc-4ac8-8bca-495d2234bb3f-openshift-service-ca\") pod \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\" (UID: \"5365d238-b3fc-4ac8-8bca-495d2234bb3f\") " Jul 22 13:14:33.755037 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.754986 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5365d238-b3fc-4ac8-8bca-495d2234bb3f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5365d238-b3fc-4ac8-8bca-495d2234bb3f" (UID: "5365d238-b3fc-4ac8-8bca-495d2234bb3f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:14:33.756681 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.756661 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5365d238-b3fc-4ac8-8bca-495d2234bb3f-kube-api-access-wh9dv" (OuterVolumeSpecName: "kube-api-access-wh9dv") pod "5365d238-b3fc-4ac8-8bca-495d2234bb3f" (UID: "5365d238-b3fc-4ac8-8bca-495d2234bb3f"). InnerVolumeSpecName "kube-api-access-wh9dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:14:33.756778 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.756756 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5365d238-b3fc-4ac8-8bca-495d2234bb3f" (UID: "5365d238-b3fc-4ac8-8bca-495d2234bb3f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:14:33.855804 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.855750 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh9dv\" (UniqueName: \"kubernetes.io/projected/5365d238-b3fc-4ac8-8bca-495d2234bb3f-kube-api-access-wh9dv\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:14:33.855804 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.855774 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5365d238-b3fc-4ac8-8bca-495d2234bb3f-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:14:33.855804 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:33.855784 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5365d238-b3fc-4ac8-8bca-495d2234bb3f-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:14:34.041066 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.041036 2576 generic.go:356] "Generic (PLEG): container finished" podID="5365d238-b3fc-4ac8-8bca-495d2234bb3f" containerID="f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5" exitCode=0 Jul 22 13:14:34.041232 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.041069 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" Jul 22 13:14:34.041232 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.041099 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" event={"ID":"5365d238-b3fc-4ac8-8bca-495d2234bb3f","Type":"ContainerDied","Data":"f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5"} Jul 22 13:14:34.041232 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.041155 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8" event={"ID":"5365d238-b3fc-4ac8-8bca-495d2234bb3f","Type":"ContainerDied","Data":"879a97eceed98dedb9d41b1551d1722b2dbe0f122e94269e5b588e9507dea866"} Jul 22 13:14:34.041232 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.041185 2576 scope.go:117] "RemoveContainer" containerID="f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5" Jul 22 13:14:34.050452 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.050438 2576 scope.go:117] "RemoveContainer" containerID="f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5" Jul 22 13:14:34.050687 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:14:34.050669 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5\": container with ID starting with f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5 not found: ID does not exist" containerID="f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5" Jul 22 13:14:34.050745 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.050698 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5"} err="failed to get container status \"f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5\": rpc error: code = NotFound desc = could not find container \"f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5\": container with ID starting with f538e8f99f4f23a28ba570b15e81c0c1bbb0337f5621a723f8e0153cc805c4e5 not found: ID does not exist" Jul 22 13:14:34.064803 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.064779 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8"] Jul 22 13:14:34.067023 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.066994 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7caca13d-66794f487-jc5s8"] Jul 22 13:14:34.713378 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:34.713345 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5365d238-b3fc-4ac8-8bca-495d2234bb3f" path="/var/lib/kubelet/pods/5365d238-b3fc-4ac8-8bca-495d2234bb3f/volumes" Jul 22 13:14:41.497167 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:41.497132 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:14:46.856178 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:14:46.856150 2576 scope.go:117] "RemoveContainer" containerID="3860273fd9d97237063f0708ba2815a05e7878d39fd19457ab24eb7e490d5f62" Jul 22 13:15:35.383082 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:35.382990 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst"] Jul 22 13:15:35.383658 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:35.383219 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" containerID="cri-o://719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9" gracePeriod=30 Jul 22 13:15:39.263370 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:39.263335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:15:40.644806 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.644785 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:15:40.797670 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.797631 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-certs\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.797670 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.797672 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-token\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.797857 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.797697 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snrl4\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-kube-api-access-snrl4\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.797893 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.797865 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istiod-ca-cert\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.797927 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.797900 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-data\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.797927 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.797916 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-credential-socket\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.798115 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.797950 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-podinfo\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.798115 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.797954 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:40.798243 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798092 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-socket\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.798243 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798154 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-envoy\") pod \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\" (UID: \"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f\") " Jul 22 13:15:40.798351 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798227 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:40.798351 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798263 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:15:40.798464 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798427 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:40.798711 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798690 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-certs\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:40.798777 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798716 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istiod-ca-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:40.798777 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798729 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-credential-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:40.798777 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798744 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-workload-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:40.799113 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.798928 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:40.800030 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.799988 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:15:40.800142 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.800032 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-kube-api-access-snrl4" (OuterVolumeSpecName: "kube-api-access-snrl4") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "kube-api-access-snrl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:15:40.800142 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.800045 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:15:40.800401 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.800388 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" (UID: "cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:15:40.899789 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.899762 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-envoy\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:40.899789 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.899790 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:40.899923 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.899804 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snrl4\" (UniqueName: \"kubernetes.io/projected/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-kube-api-access-snrl4\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:40.899923 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.899817 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-data\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:40.899923 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:40.899831 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f-istio-podinfo\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:15:41.396340 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.396311 2576 generic.go:356] "Generic (PLEG): container finished" podID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerID="719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9" exitCode=0 Jul 22 13:15:41.396492 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.396361 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" Jul 22 13:15:41.396492 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.396395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" event={"ID":"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f","Type":"ContainerDied","Data":"719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9"} Jul 22 13:15:41.396492 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.396445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst" event={"ID":"cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f","Type":"ContainerDied","Data":"33df8cc9cc3cc56f09a8ae6802fdf15f4bf110d3cc4f0852c24e8d4885c207f7"} Jul 22 13:15:41.396492 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.396467 2576 scope.go:117] "RemoveContainer" containerID="719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9" Jul 22 13:15:41.408357 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.408341 2576 scope.go:117] "RemoveContainer" containerID="1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b" Jul 22 13:15:41.416171 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.416157 2576 scope.go:117] "RemoveContainer" containerID="719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9" Jul 22 13:15:41.416405 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:15:41.416389 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9\": container with ID starting with 719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9 not found: ID does not exist" containerID="719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9" Jul 22 13:15:41.416452 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.416414 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9"} err="failed to get container status \"719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9\": rpc error: code = NotFound desc = could not find container \"719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9\": container with ID starting with 719ffa4669fc370707f16c183ff23be8a7cb1db7298248440314f82617b6e8f9 not found: ID does not exist" Jul 22 13:15:41.416452 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.416431 2576 scope.go:117] "RemoveContainer" containerID="1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b" Jul 22 13:15:41.416654 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:15:41.416635 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b\": container with ID starting with 1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b not found: ID does not exist" containerID="1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b" Jul 22 13:15:41.416698 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.416661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b"} err="failed to get container status \"1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b\": rpc error: code = NotFound desc = could not find container \"1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b\": container with ID starting with 1fcdd7925b1aaf71a0d9ad35c7fa5c403466a1c510ea63963a42ebcc110fb57b not found: ID does not exist" Jul 22 13:15:41.419844 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.419824 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst"] Jul 22 13:15:41.423502 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:41.423483 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7caca13d-openshift-default-b9c66d967-g8xst"] Jul 22 13:15:42.712589 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:42.712555 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" path="/var/lib/kubelet/pods/cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f/volumes" Jul 22 13:15:46.787420 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:46.787390 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:15:46.794824 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:46.794806 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:15:46.795440 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:46.795417 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:15:46.802599 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:46.802585 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:15:46.879322 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:46.879300 2576 scope.go:117] "RemoveContainer" containerID="7c47c7f84df60de64ffde2b2e5bae6edbff259046bb1995f4d0ab90acc13e621" Jul 22 13:15:48.912871 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:15:48.912830 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:16:32.748108 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748046 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff"] Jul 22 13:16:32.748675 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748580 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" Jul 22 13:16:32.748675 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748597 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" Jul 22 13:16:32.748675 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748611 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5365d238-b3fc-4ac8-8bca-495d2234bb3f" containerName="maas-api" Jul 22 13:16:32.748675 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748618 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5365d238-b3fc-4ac8-8bca-495d2234bb3f" containerName="maas-api" Jul 22 13:16:32.748675 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748633 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" Jul 22 13:16:32.748675 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748640 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" Jul 22 13:16:32.749061 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748699 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" Jul 22 13:16:32.749061 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.748709 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5365d238-b3fc-4ac8-8bca-495d2234bb3f" containerName="maas-api" Jul 22 13:16:32.751781 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.751764 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.755312 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.755297 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-50f07d-openshift-default-dockercfg-xwkzw\"" Jul 22 13:16:32.767746 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.767714 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff"] Jul 22 13:16:32.847046 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.846996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-certs\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.847238 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.847053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-token\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.847238 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.847102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-podinfo\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.847238 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.847188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istiod-ca-cert\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.847238 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.847211 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-credential-socket\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.847468 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.847354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-data\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.847468 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.847392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zcg9\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-kube-api-access-9zcg9\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.847468 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.847413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-envoy\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.847468 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.847437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-socket\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948142 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-data\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948142 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948142 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zcg9\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-kube-api-access-9zcg9\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948403 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-envoy\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948403 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-socket\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948403 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-certs\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948403 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-token\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948622 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-podinfo\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948622 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istiod-ca-cert\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948622 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-credential-socket\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948781 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-data\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948838 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-socket\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.948973 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.948951 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-credential-socket\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.949113 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.949090 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-certs\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.949310 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.949283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istiod-ca-cert\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.950629 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.950609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-envoy\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.950895 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.950877 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-podinfo\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.957481 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.957457 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-token\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:32.958550 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:32.958529 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zcg9\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-kube-api-access-9zcg9\") pod \"e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:33.061974 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:33.061871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:33.205886 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:33.205856 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff"] Jul 22 13:16:33.206553 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:16:33.206526 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a941e3_f38d_4c24_8f2a_13b02cebd5ac.slice/crio-2404469997a850efb6d6073ad098519a486e6d66df29683c1eb76f8fea1b5cde WatchSource:0}: Error finding container 2404469997a850efb6d6073ad098519a486e6d66df29683c1eb76f8fea1b5cde: Status 404 returned error can't find the container with id 2404469997a850efb6d6073ad098519a486e6d66df29683c1eb76f8fea1b5cde Jul 22 13:16:33.208761 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:33.208730 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:16:33.208830 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:33.208801 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:16:33.208867 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:33.208831 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:16:33.677582 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:33.677553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" event={"ID":"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac","Type":"ContainerStarted","Data":"17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532"} Jul 22 13:16:33.677769 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:33.677603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" event={"ID":"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac","Type":"ContainerStarted","Data":"2404469997a850efb6d6073ad098519a486e6d66df29683c1eb76f8fea1b5cde"} Jul 22 13:16:33.705586 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:33.705536 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" podStartSLOduration=1.7055198169999999 podStartE2EDuration="1.705519817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:16:33.701722408 +0000 UTC m=+2147.506439304" watchObservedRunningTime="2026-07-22 13:16:33.705519817 +0000 UTC m=+2147.510236712" Jul 22 13:16:34.062845 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:34.062765 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:34.067726 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:34.067703 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:34.683265 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:34.683235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:34.684118 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:34.684100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:35.895149 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:35.895118 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb"] Jul 22 13:16:35.895594 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:35.895579 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3d3b8f-ce7e-4721-a5b0-90c2ad63d10f" containerName="istio-proxy" Jul 22 13:16:35.898832 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:35.898810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:35.903399 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:35.903372 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-50f07d-serving-cert\"" Jul 22 13:16:35.913657 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:35.913633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb"] Jul 22 13:16:36.075822 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.075769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93301d1d-74c4-46ea-b687-598b095c55df-maas-api-tls\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.076048 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.075909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93301d1d-74c4-46ea-b687-598b095c55df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.076048 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.075969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nk2t\" (UniqueName: \"kubernetes.io/projected/93301d1d-74c4-46ea-b687-598b095c55df-kube-api-access-9nk2t\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.176746 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.176716 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93301d1d-74c4-46ea-b687-598b095c55df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.176938 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.176760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nk2t\" (UniqueName: \"kubernetes.io/projected/93301d1d-74c4-46ea-b687-598b095c55df-kube-api-access-9nk2t\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.176938 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.176813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93301d1d-74c4-46ea-b687-598b095c55df-maas-api-tls\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.177410 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.177388 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93301d1d-74c4-46ea-b687-598b095c55df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.179247 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.179226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93301d1d-74c4-46ea-b687-598b095c55df-maas-api-tls\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.185146 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.185125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nk2t\" (UniqueName: \"kubernetes.io/projected/93301d1d-74c4-46ea-b687-598b095c55df-kube-api-access-9nk2t\") pod \"maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.211138 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.211113 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:36.356752 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.356712 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb"] Jul 22 13:16:36.359842 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:16:36.359815 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93301d1d_74c4_46ea_b687_598b095c55df.slice/crio-3aaf30d1c9b0e5f6e0306a8bc0f746a0f0e930533ee23b25bb44c10b7a872dd1 WatchSource:0}: Error finding container 3aaf30d1c9b0e5f6e0306a8bc0f746a0f0e930533ee23b25bb44c10b7a872dd1: Status 404 returned error can't find the container with id 3aaf30d1c9b0e5f6e0306a8bc0f746a0f0e930533ee23b25bb44c10b7a872dd1 Jul 22 13:16:36.696304 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:36.696271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" event={"ID":"93301d1d-74c4-46ea-b687-598b095c55df","Type":"ContainerStarted","Data":"3aaf30d1c9b0e5f6e0306a8bc0f746a0f0e930533ee23b25bb44c10b7a872dd1"} Jul 22 13:16:37.703538 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:37.703503 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" event={"ID":"93301d1d-74c4-46ea-b687-598b095c55df","Type":"ContainerStarted","Data":"4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4"} Jul 22 13:16:37.703919 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:37.703617 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:37.720325 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:37.720282 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" podStartSLOduration=2.312846308 podStartE2EDuration="2.72026872s" totalImagesPullingTime="407.422412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:16:37.719474702 +0000 UTC m=+2151.524191601" watchObservedRunningTime="2026-07-22 13:16:37.72026872 +0000 UTC m=+2151.524985614" Jul 22 13:16:40.408156 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:40.408121 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:16:43.724163 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:43.724128 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:16:44.484517 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.484483 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd"] Jul 22 13:16:44.489318 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.489296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:44.491813 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.491795 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ab0ebe-serving-cert\"" Jul 22 13:16:44.497825 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.497801 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd"] Jul 22 13:16:44.648576 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.648542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b49636-9a42-44e9-bb64-fbcc61b4b621-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:44.648803 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.648599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:44.648803 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.648642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbhql\" (UniqueName: \"kubernetes.io/projected/33b49636-9a42-44e9-bb64-fbcc61b4b621-kube-api-access-xbhql\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:44.749273 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.749198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:44.749273 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.749232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbhql\" (UniqueName: \"kubernetes.io/projected/33b49636-9a42-44e9-bb64-fbcc61b4b621-kube-api-access-xbhql\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:44.749797 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.749303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b49636-9a42-44e9-bb64-fbcc61b4b621-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:44.749797 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:16:44.749345 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-serving-cert: secret "maas-api-e2e-inf-b-ab0ebe-serving-cert" not found Jul 22 13:16:44.749797 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:16:44.749408 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls podName:33b49636-9a42-44e9-bb64-fbcc61b4b621 nodeName:}" failed. No retries permitted until 2026-07-22 13:16:45.249393459 +0000 UTC m=+2159.054110334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls") pod "maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" (UID: "33b49636-9a42-44e9-bb64-fbcc61b4b621") : secret "maas-api-e2e-inf-b-ab0ebe-serving-cert" not found Jul 22 13:16:44.749965 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.749928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b49636-9a42-44e9-bb64-fbcc61b4b621-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:44.760661 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:44.760632 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbhql\" (UniqueName: \"kubernetes.io/projected/33b49636-9a42-44e9-bb64-fbcc61b4b621-kube-api-access-xbhql\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:45.253991 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:45.253953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:45.256777 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:45.256749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls\") pod \"maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:45.433858 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:45.433819 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:45.597838 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:45.597811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd"] Jul 22 13:16:45.748317 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:45.748284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" event={"ID":"33b49636-9a42-44e9-bb64-fbcc61b4b621","Type":"ContainerStarted","Data":"cdb2fdb4721a283fc0953e10bfebc9a1c39f184e559e9647a1bd63ae38d6ff5a"} Jul 22 13:16:46.754782 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:46.754742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" event={"ID":"33b49636-9a42-44e9-bb64-fbcc61b4b621","Type":"ContainerStarted","Data":"b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3"} Jul 22 13:16:46.755228 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:46.754868 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:46.774851 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:46.774794 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" podStartSLOduration=2.412092507 podStartE2EDuration="2.774773364s" totalImagesPullingTime="362.680857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:16:46.771937765 +0000 UTC m=+2160.576654653" watchObservedRunningTime="2026-07-22 13:16:46.774773364 +0000 UTC m=+2160.579490286" Jul 22 13:16:48.766740 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:48.766654 2576 generic.go:356] "Generic (PLEG): container finished" podID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerID="17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532" exitCode=0 Jul 22 13:16:48.767180 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:48.766724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" event={"ID":"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac","Type":"ContainerDied","Data":"17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532"} Jul 22 13:16:48.767180 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:48.767139 2576 scope.go:117] "RemoveContainer" containerID="17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532" Jul 22 13:16:48.771658 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:48.771623 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:16:48.771782 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:48.771704 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:16:48.771782 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:48.771750 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:16:49.683205 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:49.683175 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:49.773987 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:49.773953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" event={"ID":"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac","Type":"ContainerStarted","Data":"6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0"} Jul 22 13:16:50.062600 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:50.062509 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:50.063762 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:50.063734 2576 patch_prober.go:28] interesting pod/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 22 13:16:50.063883 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:50.063783 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 22 13:16:51.062966 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:51.062929 2576 patch_prober.go:28] interesting pod/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 22 13:16:51.063389 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:51.062989 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 22 13:16:51.688497 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:51.688459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:16:52.062606 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:52.062518 2576 patch_prober.go:28] interesting pod/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 22 13:16:52.062606 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:52.062571 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 22 13:16:52.765812 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:52.765786 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:16:53.062572 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:53.062475 2576 patch_prober.go:28] interesting pod/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 22 13:16:53.062572 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:53.062538 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 22 13:16:54.066425 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:54.066396 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:54.066814 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:54.066676 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:16:54.067357 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:16:54.067339 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:17:05.593908 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:17:05.593873 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:17:24.414441 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:17:24.414406 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:18:02.898776 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:02.898743 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:18:09.901665 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:09.901478 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb"] Jul 22 13:18:09.902238 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:09.901796 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" podUID="93301d1d-74c4-46ea-b687-598b095c55df" containerName="maas-api" containerID="cri-o://4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4" gracePeriod=30 Jul 22 13:18:10.171658 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.171628 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:18:10.202553 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.202518 2576 generic.go:356] "Generic (PLEG): container finished" podID="93301d1d-74c4-46ea-b687-598b095c55df" containerID="4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4" exitCode=0 Jul 22 13:18:10.202716 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.202566 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" Jul 22 13:18:10.202716 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.202576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" event={"ID":"93301d1d-74c4-46ea-b687-598b095c55df","Type":"ContainerDied","Data":"4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4"} Jul 22 13:18:10.202716 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.202646 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb" event={"ID":"93301d1d-74c4-46ea-b687-598b095c55df","Type":"ContainerDied","Data":"3aaf30d1c9b0e5f6e0306a8bc0f746a0f0e930533ee23b25bb44c10b7a872dd1"} Jul 22 13:18:10.202716 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.202681 2576 scope.go:117] "RemoveContainer" containerID="4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4" Jul 22 13:18:10.211593 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.211578 2576 scope.go:117] "RemoveContainer" containerID="4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4" Jul 22 13:18:10.211847 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:18:10.211831 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4\": container with ID starting with 4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4 not found: ID does not exist" containerID="4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4" Jul 22 13:18:10.211914 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.211858 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4"} err="failed to get container status \"4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4\": rpc error: code = NotFound desc = could not find container \"4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4\": container with ID starting with 4db65ff6a688a33cc5384577501f9c2532b2bcdd485bce3e1b599b043422cec4 not found: ID does not exist" Jul 22 13:18:10.238175 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.238152 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93301d1d-74c4-46ea-b687-598b095c55df-maas-api-tls\") pod \"93301d1d-74c4-46ea-b687-598b095c55df\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " Jul 22 13:18:10.238277 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.238262 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93301d1d-74c4-46ea-b687-598b095c55df-openshift-service-ca\") pod \"93301d1d-74c4-46ea-b687-598b095c55df\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " Jul 22 13:18:10.238344 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.238300 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nk2t\" (UniqueName: \"kubernetes.io/projected/93301d1d-74c4-46ea-b687-598b095c55df-kube-api-access-9nk2t\") pod \"93301d1d-74c4-46ea-b687-598b095c55df\" (UID: \"93301d1d-74c4-46ea-b687-598b095c55df\") " Jul 22 13:18:10.238597 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.238575 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93301d1d-74c4-46ea-b687-598b095c55df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93301d1d-74c4-46ea-b687-598b095c55df" (UID: "93301d1d-74c4-46ea-b687-598b095c55df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:18:10.240209 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.240188 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93301d1d-74c4-46ea-b687-598b095c55df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93301d1d-74c4-46ea-b687-598b095c55df" (UID: "93301d1d-74c4-46ea-b687-598b095c55df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:18:10.240426 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.240404 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93301d1d-74c4-46ea-b687-598b095c55df-kube-api-access-9nk2t" (OuterVolumeSpecName: "kube-api-access-9nk2t") pod "93301d1d-74c4-46ea-b687-598b095c55df" (UID: "93301d1d-74c4-46ea-b687-598b095c55df"). InnerVolumeSpecName "kube-api-access-9nk2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:18:10.339467 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.339446 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93301d1d-74c4-46ea-b687-598b095c55df-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:10.339467 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.339467 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nk2t\" (UniqueName: \"kubernetes.io/projected/93301d1d-74c4-46ea-b687-598b095c55df-kube-api-access-9nk2t\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:10.339589 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.339477 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93301d1d-74c4-46ea-b687-598b095c55df-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:10.525440 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.525420 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb"] Jul 22 13:18:10.531250 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.531226 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-50f07d-b5575df64-gkcgb"] Jul 22 13:18:10.712698 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:10.712673 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93301d1d-74c4-46ea-b687-598b095c55df" path="/var/lib/kubelet/pods/93301d1d-74c4-46ea-b687-598b095c55df/volumes" Jul 22 13:18:15.695894 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:15.695859 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:18:21.550192 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:21.550160 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff"] Jul 22 13:18:21.550579 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:21.550379 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" containerID="cri-o://6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0" gracePeriod=30 Jul 22 13:18:24.069128 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:24.069090 2576 patch_prober.go:28] interesting pod/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 13:18:24.069494 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:24.069141 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 13:18:26.806395 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.806372 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:18:26.874571 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874495 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-certs\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874571 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874527 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-data\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874571 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874545 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-envoy\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874571 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874567 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-token\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874888 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874600 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-credential-socket\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874888 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874629 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zcg9\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-kube-api-access-9zcg9\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874888 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874653 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-socket\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874888 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874687 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-podinfo\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874888 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istiod-ca-cert\") pod \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\" (UID: \"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac\") " Jul 22 13:18:26.874888 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874813 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:26.875228 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874950 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:26.875228 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.874966 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:26.875228 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.875087 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:26.875228 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.875104 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-workload-certs\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:26.875228 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.875161 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:18:26.875489 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.875469 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:26.876760 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.876711 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:26.877133 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.877107 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-kube-api-access-9zcg9" (OuterVolumeSpecName: "kube-api-access-9zcg9") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "kube-api-access-9zcg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:18:26.877325 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.877304 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:18:26.877424 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.877408 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" (UID: "a6a941e3-f38d-4c24-8f2a-13b02cebd5ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:18:26.975650 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.975628 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-data\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:26.975650 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.975647 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-envoy\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:26.975650 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.975655 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:26.975812 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.975663 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-credential-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:26.975812 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.975672 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zcg9\" (UniqueName: \"kubernetes.io/projected/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-kube-api-access-9zcg9\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:26.975812 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.975680 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istio-podinfo\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:26.975812 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:26.975687 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac-istiod-ca-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:27.293191 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.293158 2576 generic.go:356] "Generic (PLEG): container finished" podID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerID="6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0" exitCode=0 Jul 22 13:18:27.293383 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.293216 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" Jul 22 13:18:27.293383 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.293250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" event={"ID":"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac","Type":"ContainerDied","Data":"6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0"} Jul 22 13:18:27.293383 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.293349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff" event={"ID":"a6a941e3-f38d-4c24-8f2a-13b02cebd5ac","Type":"ContainerDied","Data":"2404469997a850efb6d6073ad098519a486e6d66df29683c1eb76f8fea1b5cde"} Jul 22 13:18:27.293383 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.293379 2576 scope.go:117] "RemoveContainer" containerID="6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0" Jul 22 13:18:27.302628 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.302614 2576 scope.go:117] "RemoveContainer" containerID="17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532" Jul 22 13:18:27.310657 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.310646 2576 scope.go:117] "RemoveContainer" containerID="6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0" Jul 22 13:18:27.310882 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:18:27.310866 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0\": container with ID starting with 6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0 not found: ID does not exist" containerID="6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0" Jul 22 13:18:27.310920 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.310890 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0"} err="failed to get container status \"6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0\": rpc error: code = NotFound desc = could not find container \"6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0\": container with ID starting with 6e252260f565961a4f02569c336a635fc2d66d8d1e46cf29974b255fbb9950e0 not found: ID does not exist" Jul 22 13:18:27.310920 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.310906 2576 scope.go:117] "RemoveContainer" containerID="17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532" Jul 22 13:18:27.311147 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:18:27.311133 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532\": container with ID starting with 17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532 not found: ID does not exist" containerID="17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532" Jul 22 13:18:27.311193 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.311152 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532"} err="failed to get container status \"17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532\": rpc error: code = NotFound desc = could not find container \"17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532\": container with ID starting with 17df696721008d13f3481a99e6aec8adf49e08c91994e2b97f9d0bab4196d532 not found: ID does not exist" Jul 22 13:18:27.314904 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.314885 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff"] Jul 22 13:18:27.319102 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:27.319076 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-50f07d-openshift-default-584c77495c-pp4ff"] Jul 22 13:18:28.713294 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:28.713255 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" path="/var/lib/kubelet/pods/a6a941e3-f38d-4c24-8f2a-13b02cebd5ac/volumes" Jul 22 13:18:32.405689 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.405655 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd"] Jul 22 13:18:32.406198 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.405962 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" podUID="33b49636-9a42-44e9-bb64-fbcc61b4b621" containerName="maas-api" containerID="cri-o://b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3" gracePeriod=30 Jul 22 13:18:32.648305 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.648280 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:18:32.730627 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.730600 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls\") pod \"33b49636-9a42-44e9-bb64-fbcc61b4b621\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " Jul 22 13:18:32.730792 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.730633 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbhql\" (UniqueName: \"kubernetes.io/projected/33b49636-9a42-44e9-bb64-fbcc61b4b621-kube-api-access-xbhql\") pod \"33b49636-9a42-44e9-bb64-fbcc61b4b621\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " Jul 22 13:18:32.730792 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.730677 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b49636-9a42-44e9-bb64-fbcc61b4b621-openshift-service-ca\") pod \"33b49636-9a42-44e9-bb64-fbcc61b4b621\" (UID: \"33b49636-9a42-44e9-bb64-fbcc61b4b621\") " Jul 22 13:18:32.731084 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.731062 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b49636-9a42-44e9-bb64-fbcc61b4b621-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33b49636-9a42-44e9-bb64-fbcc61b4b621" (UID: "33b49636-9a42-44e9-bb64-fbcc61b4b621"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:18:32.732854 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.732833 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33b49636-9a42-44e9-bb64-fbcc61b4b621" (UID: "33b49636-9a42-44e9-bb64-fbcc61b4b621"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:18:32.732917 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.732889 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b49636-9a42-44e9-bb64-fbcc61b4b621-kube-api-access-xbhql" (OuterVolumeSpecName: "kube-api-access-xbhql") pod "33b49636-9a42-44e9-bb64-fbcc61b4b621" (UID: "33b49636-9a42-44e9-bb64-fbcc61b4b621"). InnerVolumeSpecName "kube-api-access-xbhql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:18:32.831325 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.831295 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33b49636-9a42-44e9-bb64-fbcc61b4b621-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:32.831325 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.831319 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbhql\" (UniqueName: \"kubernetes.io/projected/33b49636-9a42-44e9-bb64-fbcc61b4b621-kube-api-access-xbhql\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:32.831325 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:32.831329 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b49636-9a42-44e9-bb64-fbcc61b4b621-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:18:33.326374 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.326339 2576 generic.go:356] "Generic (PLEG): container finished" podID="33b49636-9a42-44e9-bb64-fbcc61b4b621" containerID="b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3" exitCode=0 Jul 22 13:18:33.326554 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.326380 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" Jul 22 13:18:33.326554 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.326427 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" event={"ID":"33b49636-9a42-44e9-bb64-fbcc61b4b621","Type":"ContainerDied","Data":"b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3"} Jul 22 13:18:33.326554 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.326476 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd" event={"ID":"33b49636-9a42-44e9-bb64-fbcc61b4b621","Type":"ContainerDied","Data":"cdb2fdb4721a283fc0953e10bfebc9a1c39f184e559e9647a1bd63ae38d6ff5a"} Jul 22 13:18:33.326554 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.326505 2576 scope.go:117] "RemoveContainer" containerID="b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3" Jul 22 13:18:33.336116 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.336091 2576 scope.go:117] "RemoveContainer" containerID="b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3" Jul 22 13:18:33.336370 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:18:33.336352 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3\": container with ID starting with b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3 not found: ID does not exist" containerID="b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3" Jul 22 13:18:33.336439 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.336382 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3"} err="failed to get container status \"b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3\": rpc error: code = NotFound desc = could not find container \"b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3\": container with ID starting with b4c44f06c60624c3db1540fa0ea49b2a8e34416582ec13995517f22c8a2b59b3 not found: ID does not exist" Jul 22 13:18:33.347964 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.347932 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd"] Jul 22 13:18:33.352181 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:33.352159 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ab0ebe-75dc69f6db-gbzxd"] Jul 22 13:18:34.712798 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:34.712762 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b49636-9a42-44e9-bb64-fbcc61b4b621" path="/var/lib/kubelet/pods/33b49636-9a42-44e9-bb64-fbcc61b4b621/volumes" Jul 22 13:18:35.088470 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:35.088393 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:18:44.688821 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.688786 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8"] Jul 22 13:18:44.689217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689152 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93301d1d-74c4-46ea-b687-598b095c55df" containerName="maas-api" Jul 22 13:18:44.689217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689164 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="93301d1d-74c4-46ea-b687-598b095c55df" containerName="maas-api" Jul 22 13:18:44.689217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689173 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" Jul 22 13:18:44.689217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689178 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" Jul 22 13:18:44.689217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689193 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b49636-9a42-44e9-bb64-fbcc61b4b621" containerName="maas-api" Jul 22 13:18:44.689217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689199 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b49636-9a42-44e9-bb64-fbcc61b4b621" containerName="maas-api" Jul 22 13:18:44.689217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689207 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" Jul 22 13:18:44.689217 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689212 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" Jul 22 13:18:44.689462 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689280 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b49636-9a42-44e9-bb64-fbcc61b4b621" containerName="maas-api" Jul 22 13:18:44.689462 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689288 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="93301d1d-74c4-46ea-b687-598b095c55df" containerName="maas-api" Jul 22 13:18:44.689462 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689293 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" Jul 22 13:18:44.689462 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.689299 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a941e3-f38d-4c24-8f2a-13b02cebd5ac" containerName="istio-proxy" Jul 22 13:18:44.693716 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.693697 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.696190 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.696165 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f3d21e-openshift-default-dockercfg-vbz8z\"" Jul 22 13:18:44.706648 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.706623 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8"] Jul 22 13:18:44.837059 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.836996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-podinfo\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.837254 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.837087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-token\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.837254 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.837118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istiod-ca-cert\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.837254 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.837146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-credential-socket\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.837254 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.837171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-certs\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.837254 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.837200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-data\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.837456 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.837284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhml\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-kube-api-access-wlhml\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.837456 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.837315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-envoy\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.837456 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.837345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-socket\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938328 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhml\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-kube-api-access-wlhml\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938328 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-envoy\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938578 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-socket\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938578 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-podinfo\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938578 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-token\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938578 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istiod-ca-cert\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938578 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-credential-socket\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938578 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-certs\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938578 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-data\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.938952 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.938891 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-socket\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.939330 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.939001 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-data\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.939330 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.939069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-credential-socket\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.939330 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.939097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-certs\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.939330 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.939253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istiod-ca-cert\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.940524 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.940504 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-envoy\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.940813 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.940795 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-podinfo\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.946658 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.946630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-token\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:44.946743 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:44.946670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhml\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-kube-api-access-wlhml\") pod \"e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:45.004589 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:45.004563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:45.143713 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:45.143691 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8"] Jul 22 13:18:45.144520 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:18:45.144495 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6a23dd_4caa_4ebf_ad23_3885098ec781.slice/crio-cdb8c5508e953f0fbeac4915947bcd8ac794021bde69d76472349faad8d94c59 WatchSource:0}: Error finding container cdb8c5508e953f0fbeac4915947bcd8ac794021bde69d76472349faad8d94c59: Status 404 returned error can't find the container with id cdb8c5508e953f0fbeac4915947bcd8ac794021bde69d76472349faad8d94c59 Jul 22 13:18:45.146615 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:45.146583 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:18:45.146687 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:45.146646 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:18:45.146687 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:45.146676 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:18:45.390861 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:45.390826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" event={"ID":"8e6a23dd-4caa-4ebf-ad23-3885098ec781","Type":"ContainerStarted","Data":"52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06"} Jul 22 13:18:45.391045 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:45.390887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" event={"ID":"8e6a23dd-4caa-4ebf-ad23-3885098ec781","Type":"ContainerStarted","Data":"cdb8c5508e953f0fbeac4915947bcd8ac794021bde69d76472349faad8d94c59"} Jul 22 13:18:45.415571 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:45.415517 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podStartSLOduration=1.415497592 podStartE2EDuration="1.415497592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:18:45.413981976 +0000 UTC m=+2279.218698874" watchObservedRunningTime="2026-07-22 13:18:45.415497592 +0000 UTC m=+2279.220214471" Jul 22 13:18:46.005750 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:46.005719 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:46.007387 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:46.007358 2576 patch_prober.go:28] interesting pod/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 22 13:18:46.007505 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:46.007408 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 22 13:18:47.009376 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.009354 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:47.401474 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.401388 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:47.402341 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.402320 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:18:47.461136 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.461104 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d"] Jul 22 13:18:47.467071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.467045 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:47.469473 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.469448 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f3d21e-serving-cert\"" Jul 22 13:18:47.474837 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.474809 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d"] Jul 22 13:18:47.562633 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.562605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:47.562786 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.562764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:47.562849 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.562831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8hk\" (UniqueName: \"kubernetes.io/projected/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-kube-api-access-lx8hk\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:47.663477 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.663450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:47.663604 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.663489 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8hk\" (UniqueName: \"kubernetes.io/projected/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-kube-api-access-lx8hk\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:47.663604 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.663519 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:47.663685 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:18:47.663610 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-serving-cert: secret "maas-api-e2e-api-a-f3d21e-serving-cert" not found Jul 22 13:18:47.663685 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:18:47.663669 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls podName:50c26009-2903-4ec9-b4fc-ef6dbf6a88a5 nodeName:}" failed. No retries permitted until 2026-07-22 13:18:48.163653989 +0000 UTC m=+2281.968370866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls") pod "maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" (UID: "50c26009-2903-4ec9-b4fc-ef6dbf6a88a5") : secret "maas-api-e2e-api-a-f3d21e-serving-cert" not found Jul 22 13:18:47.664164 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.664146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:47.673053 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:47.673034 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8hk\" (UniqueName: \"kubernetes.io/projected/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-kube-api-access-lx8hk\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:48.167595 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:48.167563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:48.170129 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:48.170107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls\") pod \"maas-api-e2e-api-a-f3d21e-8497697858-bdm7d\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:48.379466 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:48.379435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:48.542125 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:48.542096 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d"] Jul 22 13:18:48.543808 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:18:48.543781 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c26009_2903_4ec9_b4fc_ef6dbf6a88a5.slice/crio-fa539d415ea3fdc744daadafb36a47c545b6ce0ffd3ac43214bd95b0ce451377 WatchSource:0}: Error finding container fa539d415ea3fdc744daadafb36a47c545b6ce0ffd3ac43214bd95b0ce451377: Status 404 returned error can't find the container with id fa539d415ea3fdc744daadafb36a47c545b6ce0ffd3ac43214bd95b0ce451377 Jul 22 13:18:48.545531 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:48.545512 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 13:18:49.414893 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:49.414858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" event={"ID":"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5","Type":"ContainerStarted","Data":"891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae"} Jul 22 13:18:49.415339 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:49.414920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" event={"ID":"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5","Type":"ContainerStarted","Data":"fa539d415ea3fdc744daadafb36a47c545b6ce0ffd3ac43214bd95b0ce451377"} Jul 22 13:18:49.434227 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:49.434188 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" podStartSLOduration=1.9446362270000002 podStartE2EDuration="2.434175431s" totalImagesPullingTime="489.539204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:18:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:18:49.43310242 +0000 UTC m=+2283.237819317" watchObservedRunningTime="2026-07-22 13:18:49.434175431 +0000 UTC m=+2283.238892326" Jul 22 13:18:50.420161 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:50.420131 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:54.292172 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:54.292137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:18:56.174145 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.174108 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk"] Jul 22 13:18:56.177840 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.177823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.180498 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.180474 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-5886b7-serving-cert\"" Jul 22 13:18:56.197766 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.197741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk"] Jul 22 13:18:56.248085 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.248054 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f35eda9-8256-42e5-a296-9ce58cdee643-openshift-service-ca\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.248215 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.248097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f35eda9-8256-42e5-a296-9ce58cdee643-maas-api-tls\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.248215 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.248144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzwfg\" (UniqueName: \"kubernetes.io/projected/2f35eda9-8256-42e5-a296-9ce58cdee643-kube-api-access-tzwfg\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.349453 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.349416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f35eda9-8256-42e5-a296-9ce58cdee643-openshift-service-ca\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.349615 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.349470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f35eda9-8256-42e5-a296-9ce58cdee643-maas-api-tls\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.349615 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.349491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzwfg\" (UniqueName: \"kubernetes.io/projected/2f35eda9-8256-42e5-a296-9ce58cdee643-kube-api-access-tzwfg\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.350094 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.350075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f35eda9-8256-42e5-a296-9ce58cdee643-openshift-service-ca\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.352246 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.352221 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f35eda9-8256-42e5-a296-9ce58cdee643-maas-api-tls\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.358158 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.358134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzwfg\" (UniqueName: \"kubernetes.io/projected/2f35eda9-8256-42e5-a296-9ce58cdee643-kube-api-access-tzwfg\") pod \"maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.431987 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.431925 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:18:56.488389 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.488359 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:56.678679 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:56.678646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk"] Jul 22 13:18:57.458861 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:57.458826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" event={"ID":"2f35eda9-8256-42e5-a296-9ce58cdee643","Type":"ContainerStarted","Data":"33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4"} Jul 22 13:18:57.459250 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:57.458879 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" event={"ID":"2f35eda9-8256-42e5-a296-9ce58cdee643","Type":"ContainerStarted","Data":"c7e8b4f53510664deceffe547dfb7e1d3034b8cf36bdddfdf74d972fad9603e1"} Jul 22 13:18:57.459250 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:57.458933 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:18:57.474757 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:18:57.474713 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" podStartSLOduration=0.806980884 podStartE2EDuration="1.474693757s" totalImagesPullingTime="667.712873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:18:57.474241792 +0000 UTC m=+2291.278958686" watchObservedRunningTime="2026-07-22 13:18:57.474693757 +0000 UTC m=+2291.279410650" Jul 22 13:19:02.403238 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:02.403151 2576 patch_prober.go:28] interesting pod/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 22 13:19:02.403238 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:02.403207 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 22 13:19:02.489206 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:02.489178 2576 generic.go:356] "Generic (PLEG): container finished" podID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerID="52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06" exitCode=0 Jul 22 13:19:02.489371 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:02.489214 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" event={"ID":"8e6a23dd-4caa-4ebf-ad23-3885098ec781","Type":"ContainerDied","Data":"52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06"} Jul 22 13:19:02.489536 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:02.489524 2576 scope.go:117] "RemoveContainer" containerID="52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06" Jul 22 13:19:02.490520 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:02.490490 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:19:02.490584 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:02.490568 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:19:02.490630 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:02.490617 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:19:03.470061 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:03.470029 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:19:03.496059 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:03.496024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" event={"ID":"8e6a23dd-4caa-4ebf-ad23-3885098ec781","Type":"ContainerStarted","Data":"3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1"} Jul 22 13:19:04.005615 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:04.005586 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:19:04.007481 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:04.007402 2576 patch_prober.go:28] interesting pod/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 22 13:19:04.007665 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:04.007644 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 22 13:19:05.005594 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:05.005560 2576 patch_prober.go:28] interesting pod/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 22 13:19:05.006161 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:05.006133 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 22 13:19:05.610192 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:05.610158 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:19:06.005665 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:06.005627 2576 patch_prober.go:28] interesting pod/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 22 13:19:06.006303 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:06.006267 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 22 13:19:07.007176 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:07.007142 2576 patch_prober.go:28] interesting pod/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 22 13:19:07.007763 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:07.007734 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 22 13:19:08.008877 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:08.008849 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:19:08.009256 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:08.009183 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:19:08.009828 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:08.009809 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:19:24.519714 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.519678 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d"] Jul 22 13:19:24.520353 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.519923 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" podUID="50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" containerName="maas-api" containerID="cri-o://891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae" gracePeriod=30 Jul 22 13:19:24.773273 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.773204 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:19:24.887104 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.887079 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls\") pod \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " Jul 22 13:19:24.887249 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.887111 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-openshift-service-ca\") pod \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " Jul 22 13:19:24.887249 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.887133 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lx8hk\" (UniqueName: \"kubernetes.io/projected/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-kube-api-access-lx8hk\") pod \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\" (UID: \"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5\") " Jul 22 13:19:24.887481 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.887448 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" (UID: "50c26009-2903-4ec9-b4fc-ef6dbf6a88a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:19:24.889314 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.889292 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-kube-api-access-lx8hk" (OuterVolumeSpecName: "kube-api-access-lx8hk") pod "50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" (UID: "50c26009-2903-4ec9-b4fc-ef6dbf6a88a5"). InnerVolumeSpecName "kube-api-access-lx8hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:19:24.889493 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.889468 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" (UID: "50c26009-2903-4ec9-b4fc-ef6dbf6a88a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:19:24.987972 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.987949 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:24.987972 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.987973 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:24.988160 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:24.987986 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8hk\" (UniqueName: \"kubernetes.io/projected/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5-kube-api-access-lx8hk\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:25.621425 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.621393 2576 generic.go:356] "Generic (PLEG): container finished" podID="50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" containerID="891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae" exitCode=0 Jul 22 13:19:25.621865 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.621437 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" Jul 22 13:19:25.621865 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.621447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" event={"ID":"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5","Type":"ContainerDied","Data":"891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae"} Jul 22 13:19:25.621865 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.621498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d" event={"ID":"50c26009-2903-4ec9-b4fc-ef6dbf6a88a5","Type":"ContainerDied","Data":"fa539d415ea3fdc744daadafb36a47c545b6ce0ffd3ac43214bd95b0ce451377"} Jul 22 13:19:25.621865 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.621530 2576 scope.go:117] "RemoveContainer" containerID="891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae" Jul 22 13:19:25.631251 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.631234 2576 scope.go:117] "RemoveContainer" containerID="891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae" Jul 22 13:19:25.631511 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:19:25.631492 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae\": container with ID starting with 891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae not found: ID does not exist" containerID="891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae" Jul 22 13:19:25.631580 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.631518 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae"} err="failed to get container status \"891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae\": rpc error: code = NotFound desc = could not find container \"891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae\": container with ID starting with 891c4e5f913602fe7fb099d8f63de1f8879e033c6f83ab5bc145eeb667cce9ae not found: ID does not exist" Jul 22 13:19:25.642634 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.642609 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d"] Jul 22 13:19:25.646473 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:25.646456 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3d21e-8497697858-bdm7d"] Jul 22 13:19:26.712460 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:26.712430 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" path="/var/lib/kubelet/pods/50c26009-2903-4ec9-b4fc-ef6dbf6a88a5/volumes" Jul 22 13:19:28.900619 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:28.900589 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:19:35.658249 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:35.658217 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8"] Jul 22 13:19:35.658654 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:35.658462 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" containerID="cri-o://3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1" gracePeriod=30 Jul 22 13:19:38.012135 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:38.012058 2576 patch_prober.go:28] interesting pod/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 22 13:19:38.012135 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:38.012119 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 13:19:40.911050 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:40.911004 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:19:41.010691 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010616 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istiod-ca-cert\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.010824 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010701 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-envoy\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.010824 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010729 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-certs\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.010824 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010758 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-socket\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.010824 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010777 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-credential-socket\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.010824 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010791 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-data\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.011097 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010815 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-token\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.011097 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010857 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlhml\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-kube-api-access-wlhml\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.011097 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.010908 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-podinfo\") pod \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\" (UID: \"8e6a23dd-4caa-4ebf-ad23-3885098ec781\") " Jul 22 13:19:41.011097 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011052 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:19:41.011097 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011070 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:19:41.011097 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011073 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:19:41.011097 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011088 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:19:41.011447 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011197 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-certs\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.011447 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011216 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-workload-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.011447 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011230 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-credential-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.011447 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011244 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istiod-ca-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.011760 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.011738 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:19:41.012903 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.012880 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:19:41.013202 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.013182 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:19:41.013527 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.013509 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:19:41.013591 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.013533 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-kube-api-access-wlhml" (OuterVolumeSpecName: "kube-api-access-wlhml") pod "8e6a23dd-4caa-4ebf-ad23-3885098ec781" (UID: "8e6a23dd-4caa-4ebf-ad23-3885098ec781"). InnerVolumeSpecName "kube-api-access-wlhml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:19:41.111661 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.111637 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-data\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.111661 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.111657 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.111792 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.111665 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlhml\" (UniqueName: \"kubernetes.io/projected/8e6a23dd-4caa-4ebf-ad23-3885098ec781-kube-api-access-wlhml\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.111792 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.111673 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-podinfo\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.111792 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.111681 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e6a23dd-4caa-4ebf-ad23-3885098ec781-istio-envoy\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:41.705043 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.704994 2576 generic.go:356] "Generic (PLEG): container finished" podID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerID="3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1" exitCode=0 Jul 22 13:19:41.705245 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.705065 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" Jul 22 13:19:41.705245 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.705089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" event={"ID":"8e6a23dd-4caa-4ebf-ad23-3885098ec781","Type":"ContainerDied","Data":"3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1"} Jul 22 13:19:41.705245 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.705141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8" event={"ID":"8e6a23dd-4caa-4ebf-ad23-3885098ec781","Type":"ContainerDied","Data":"cdb8c5508e953f0fbeac4915947bcd8ac794021bde69d76472349faad8d94c59"} Jul 22 13:19:41.705245 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.705163 2576 scope.go:117] "RemoveContainer" containerID="3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1" Jul 22 13:19:41.714546 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.714530 2576 scope.go:117] "RemoveContainer" containerID="52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06" Jul 22 13:19:41.722435 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.722420 2576 scope.go:117] "RemoveContainer" containerID="3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1" Jul 22 13:19:41.722682 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:19:41.722665 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1\": container with ID starting with 3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1 not found: ID does not exist" containerID="3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1" Jul 22 13:19:41.722730 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.722692 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1"} err="failed to get container status \"3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1\": rpc error: code = NotFound desc = could not find container \"3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1\": container with ID starting with 3665b5a0afc276675470e462aaa7ad9103724a0cedadece529f3e90785d4e8b1 not found: ID does not exist" Jul 22 13:19:41.722730 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.722709 2576 scope.go:117] "RemoveContainer" containerID="52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06" Jul 22 13:19:41.722919 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:19:41.722902 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06\": container with ID starting with 52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06 not found: ID does not exist" containerID="52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06" Jul 22 13:19:41.722971 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.722927 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06"} err="failed to get container status \"52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06\": rpc error: code = NotFound desc = could not find container \"52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06\": container with ID starting with 52408db6e8c8a9b3c968208cb18bde817686b4fa94f2f957dce2362d87c9ae06 not found: ID does not exist" Jul 22 13:19:41.726968 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.726945 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8"] Jul 22 13:19:41.731716 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:41.731692 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f3d21e-openshift-default-68b9698dc7-skkz8"] Jul 22 13:19:42.716606 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:42.716566 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" path="/var/lib/kubelet/pods/8e6a23dd-4caa-4ebf-ad23-3885098ec781/volumes" Jul 22 13:19:46.939654 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:46.939622 2576 scope.go:117] "RemoveContainer" containerID="824fda179cbdcfcfc37afe1488f4f4564cf5241c6d9f791d05d93b2195eacc7b" Jul 22 13:19:51.666704 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.666672 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk"] Jul 22 13:19:51.667197 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.666884 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" podUID="2f35eda9-8256-42e5-a296-9ce58cdee643" containerName="maas-api" containerID="cri-o://33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4" gracePeriod=30 Jul 22 13:19:51.918785 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.918730 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:19:51.993900 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.993870 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f35eda9-8256-42e5-a296-9ce58cdee643-openshift-service-ca\") pod \"2f35eda9-8256-42e5-a296-9ce58cdee643\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " Jul 22 13:19:51.994081 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.993983 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f35eda9-8256-42e5-a296-9ce58cdee643-maas-api-tls\") pod \"2f35eda9-8256-42e5-a296-9ce58cdee643\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " Jul 22 13:19:51.994081 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.994054 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzwfg\" (UniqueName: \"kubernetes.io/projected/2f35eda9-8256-42e5-a296-9ce58cdee643-kube-api-access-tzwfg\") pod \"2f35eda9-8256-42e5-a296-9ce58cdee643\" (UID: \"2f35eda9-8256-42e5-a296-9ce58cdee643\") " Jul 22 13:19:51.994199 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.994183 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f35eda9-8256-42e5-a296-9ce58cdee643-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f35eda9-8256-42e5-a296-9ce58cdee643" (UID: "2f35eda9-8256-42e5-a296-9ce58cdee643"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:19:51.994307 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.994292 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f35eda9-8256-42e5-a296-9ce58cdee643-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:51.996027 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.995996 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f35eda9-8256-42e5-a296-9ce58cdee643-kube-api-access-tzwfg" (OuterVolumeSpecName: "kube-api-access-tzwfg") pod "2f35eda9-8256-42e5-a296-9ce58cdee643" (UID: "2f35eda9-8256-42e5-a296-9ce58cdee643"). InnerVolumeSpecName "kube-api-access-tzwfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:19:51.996113 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:51.996079 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f35eda9-8256-42e5-a296-9ce58cdee643-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f35eda9-8256-42e5-a296-9ce58cdee643" (UID: "2f35eda9-8256-42e5-a296-9ce58cdee643"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:19:52.095446 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.095423 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzwfg\" (UniqueName: \"kubernetes.io/projected/2f35eda9-8256-42e5-a296-9ce58cdee643-kube-api-access-tzwfg\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:52.095446 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.095445 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f35eda9-8256-42e5-a296-9ce58cdee643-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:19:52.767520 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.767488 2576 generic.go:356] "Generic (PLEG): container finished" podID="2f35eda9-8256-42e5-a296-9ce58cdee643" containerID="33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4" exitCode=0 Jul 22 13:19:52.767887 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.767531 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" Jul 22 13:19:52.767887 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.767575 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" event={"ID":"2f35eda9-8256-42e5-a296-9ce58cdee643","Type":"ContainerDied","Data":"33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4"} Jul 22 13:19:52.767887 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.767633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk" event={"ID":"2f35eda9-8256-42e5-a296-9ce58cdee643","Type":"ContainerDied","Data":"c7e8b4f53510664deceffe547dfb7e1d3034b8cf36bdddfdf74d972fad9603e1"} Jul 22 13:19:52.767887 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.767665 2576 scope.go:117] "RemoveContainer" containerID="33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4" Jul 22 13:19:52.776334 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.776318 2576 scope.go:117] "RemoveContainer" containerID="33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4" Jul 22 13:19:52.776606 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:19:52.776579 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4\": container with ID starting with 33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4 not found: ID does not exist" containerID="33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4" Jul 22 13:19:52.776673 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.776619 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4"} err="failed to get container status \"33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4\": rpc error: code = NotFound desc = could not find container \"33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4\": container with ID starting with 33efb0aa6f4b4369c1eb319f50cfd3ec174dc2ae1cd0e181e2fa80cee3f641b4 not found: ID does not exist" Jul 22 13:19:52.784984 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.784963 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk"] Jul 22 13:19:52.789521 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:52.789498 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-5886b7-5c7fdc7966-7krbk"] Jul 22 13:19:53.798937 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:53.798904 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:19:54.712544 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:19:54.712504 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f35eda9-8256-42e5-a296-9ce58cdee643" path="/var/lib/kubelet/pods/2f35eda9-8256-42e5-a296-9ce58cdee643/volumes" Jul 22 13:20:04.088099 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088062 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb"] Jul 22 13:20:04.088589 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088569 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" Jul 22 13:20:04.088589 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088587 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" Jul 22 13:20:04.088707 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088633 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" containerName="maas-api" Jul 22 13:20:04.088707 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088643 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" containerName="maas-api" Jul 22 13:20:04.088707 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088653 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f35eda9-8256-42e5-a296-9ce58cdee643" containerName="maas-api" Jul 22 13:20:04.088707 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088662 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f35eda9-8256-42e5-a296-9ce58cdee643" containerName="maas-api" Jul 22 13:20:04.088896 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088770 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" Jul 22 13:20:04.088896 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088782 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50c26009-2903-4ec9-b4fc-ef6dbf6a88a5" containerName="maas-api" Jul 22 13:20:04.088896 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.088793 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f35eda9-8256-42e5-a296-9ce58cdee643" containerName="maas-api" Jul 22 13:20:04.093376 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.093356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.095988 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.095967 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d24327-openshift-default-dockercfg-2wqc8\"" Jul 22 13:20:04.105456 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.105432 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb"] Jul 22 13:20:04.191581 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-data\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.191759 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-istio-token\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.191759 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-certs\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.191759 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-envoy\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.191759 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c950c-6f49-4283-b36c-63313aaafa19-istio-podinfo\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.191975 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-credential-socket\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.191975 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c950c-6f49-4283-b36c-63313aaafa19-istiod-ca-cert\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.191975 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9l6w\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-kube-api-access-z9l6w\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.191975 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.191860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-socket\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292581 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292534 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9l6w\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-kube-api-access-z9l6w\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292581 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-socket\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292778 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-data\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292778 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292659 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-istio-token\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292778 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-certs\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292778 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-envoy\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292975 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c950c-6f49-4283-b36c-63313aaafa19-istio-podinfo\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292975 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-credential-socket\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.292975 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.292964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c950c-6f49-4283-b36c-63313aaafa19-istiod-ca-cert\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.293152 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.293052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-data\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.293152 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.293085 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-socket\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.293247 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.293202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-certs\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.293315 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.293296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-credential-socket\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.293621 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.293605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c950c-6f49-4283-b36c-63313aaafa19-istiod-ca-cert\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.295169 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.295142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-envoy\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.295339 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.295322 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c950c-6f49-4283-b36c-63313aaafa19-istio-podinfo\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.301241 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.301208 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9l6w\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-kube-api-access-z9l6w\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.302148 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.302125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-istio-token\") pod \"e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.404914 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.404847 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:04.542571 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.542549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb"] Jul 22 13:20:04.545154 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:20:04.545112 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065c950c_6f49_4283_b36c_63313aaafa19.slice/crio-6943a2cda2de8ff051938893905de8157af263f968616e55e6f64fc4a5ede552 WatchSource:0}: Error finding container 6943a2cda2de8ff051938893905de8157af263f968616e55e6f64fc4a5ede552: Status 404 returned error can't find the container with id 6943a2cda2de8ff051938893905de8157af263f968616e55e6f64fc4a5ede552 Jul 22 13:20:04.547307 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.547276 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:04.547374 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.547341 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:04.547413 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.547372 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:04.837590 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.837557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" event={"ID":"065c950c-6f49-4283-b36c-63313aaafa19","Type":"ContainerStarted","Data":"d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853"} Jul 22 13:20:04.837760 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.837614 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" event={"ID":"065c950c-6f49-4283-b36c-63313aaafa19","Type":"ContainerStarted","Data":"6943a2cda2de8ff051938893905de8157af263f968616e55e6f64fc4a5ede552"} Jul 22 13:20:04.863258 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:04.863211 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" podStartSLOduration=0.863196774 podStartE2EDuration="863.196774ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:20:04.861581825 +0000 UTC m=+2358.666298724" watchObservedRunningTime="2026-07-22 13:20:04.863196774 +0000 UTC m=+2358.667913669" Jul 22 13:20:05.405968 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:05.405933 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:05.411284 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:05.411263 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:05.842107 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:05.842071 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:05.842930 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:05.842912 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:07.186439 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.186408 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg"] Jul 22 13:20:07.186800 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.186761 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" Jul 22 13:20:07.186800 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.186773 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" Jul 22 13:20:07.186872 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.186839 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6a23dd-4caa-4ebf-ad23-3885098ec781" containerName="istio-proxy" Jul 22 13:20:07.189905 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.189891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.192935 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.192912 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d24327-serving-cert\"" Jul 22 13:20:07.204451 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.204426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg"] Jul 22 13:20:07.316041 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.315993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a2dfe2-597b-4832-8782-20828a052a2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.316181 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.316118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a2dfe2-597b-4832-8782-20828a052a2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.316181 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.316153 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ln9\" (UniqueName: \"kubernetes.io/projected/10a2dfe2-597b-4832-8782-20828a052a2e-kube-api-access-v8ln9\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.417082 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.417050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a2dfe2-597b-4832-8782-20828a052a2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.417241 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.417120 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a2dfe2-597b-4832-8782-20828a052a2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.417241 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.417145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ln9\" (UniqueName: \"kubernetes.io/projected/10a2dfe2-597b-4832-8782-20828a052a2e-kube-api-access-v8ln9\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.417686 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.417664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a2dfe2-597b-4832-8782-20828a052a2e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.419743 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.419713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a2dfe2-597b-4832-8782-20828a052a2e-maas-api-tls\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.430826 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.430796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ln9\" (UniqueName: \"kubernetes.io/projected/10a2dfe2-597b-4832-8782-20828a052a2e-kube-api-access-v8ln9\") pod \"maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.500576 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.500512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:07.897715 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:07.897689 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg"] Jul 22 13:20:07.898329 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:20:07.898303 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a2dfe2_597b_4832_8782_20828a052a2e.slice/crio-6f77ab02a789ec30d3d0bd21ab0ee89e84c3b7a831daf10061c48d5ee9b53d99 WatchSource:0}: Error finding container 6f77ab02a789ec30d3d0bd21ab0ee89e84c3b7a831daf10061c48d5ee9b53d99: Status 404 returned error can't find the container with id 6f77ab02a789ec30d3d0bd21ab0ee89e84c3b7a831daf10061c48d5ee9b53d99 Jul 22 13:20:08.858303 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:08.858268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" event={"ID":"10a2dfe2-597b-4832-8782-20828a052a2e","Type":"ContainerStarted","Data":"a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37"} Jul 22 13:20:08.858303 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:08.858323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" event={"ID":"10a2dfe2-597b-4832-8782-20828a052a2e","Type":"ContainerStarted","Data":"6f77ab02a789ec30d3d0bd21ab0ee89e84c3b7a831daf10061c48d5ee9b53d99"} Jul 22 13:20:08.858767 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:08.858343 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:08.890534 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:08.890485 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" podStartSLOduration=1.57364506 podStartE2EDuration="1.890471867s" totalImagesPullingTime="316.826807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:20:08.888314209 +0000 UTC m=+2362.693031105" watchObservedRunningTime="2026-07-22 13:20:08.890471867 +0000 UTC m=+2362.695188762" Jul 22 13:20:12.085266 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.085218 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd"] Jul 22 13:20:12.089524 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.089508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.092086 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.092066 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-ffc1de-openshift-default-dockercfg-g5d9r\"" Jul 22 13:20:12.100989 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.100965 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd"] Jul 22 13:20:12.261854 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.261824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqnl2\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-kube-api-access-mqnl2\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.262071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.261865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-socket\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.262071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.261892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-podinfo\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.262071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.261919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6168-7078-45a5-b040-8e5e22e7e82e-istiod-ca-cert\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.262071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.261991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-certs\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.262307 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.262107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-data\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.262307 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.262138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-token\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.262307 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.262164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-envoy\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.262307 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.262231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-credential-socket\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363357 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-credential-socket\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363357 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqnl2\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-kube-api-access-mqnl2\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363357 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-socket\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363601 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-podinfo\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363601 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6168-7078-45a5-b040-8e5e22e7e82e-istiod-ca-cert\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363601 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-certs\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363601 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-data\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363601 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-token\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363847 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363606 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-envoy\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363847 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-credential-socket\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.363951 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-certs\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.364006 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363960 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-data\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.364006 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.363955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-socket\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.364247 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.364226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6168-7078-45a5-b040-8e5e22e7e82e-istiod-ca-cert\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.365894 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.365872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-envoy\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.366036 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.366003 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-podinfo\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.372068 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.372004 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqnl2\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-kube-api-access-mqnl2\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.372349 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.372334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-token\") pod \"e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.401443 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.401419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:12.536084 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.536061 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd"] Jul 22 13:20:12.537720 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:20:12.537687 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50aa6168_7078_45a5_b040_8e5e22e7e82e.slice/crio-5dc1527463acf19f46009858795d590bac4e4aa65681fd775f0e720caca33f3b WatchSource:0}: Error finding container 5dc1527463acf19f46009858795d590bac4e4aa65681fd775f0e720caca33f3b: Status 404 returned error can't find the container with id 5dc1527463acf19f46009858795d590bac4e4aa65681fd775f0e720caca33f3b Jul 22 13:20:12.540798 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.540756 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:12.540924 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.540850 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:12.540924 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.540892 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:12.881536 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.881505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" event={"ID":"50aa6168-7078-45a5-b040-8e5e22e7e82e","Type":"ContainerStarted","Data":"a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d"} Jul 22 13:20:12.881740 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.881553 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" event={"ID":"50aa6168-7078-45a5-b040-8e5e22e7e82e","Type":"ContainerStarted","Data":"5dc1527463acf19f46009858795d590bac4e4aa65681fd775f0e720caca33f3b"} Jul 22 13:20:12.902700 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:12.902648 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" podStartSLOduration=0.902629987 podStartE2EDuration="902.629987ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:20:12.901842035 +0000 UTC m=+2366.706558946" watchObservedRunningTime="2026-07-22 13:20:12.902629987 +0000 UTC m=+2366.707346884" Jul 22 13:20:13.402369 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:13.402335 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:13.406914 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:13.406893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:13.887801 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:13.887773 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:13.888787 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:13.888767 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:13.998915 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:13.998874 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:20:14.868686 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:14.868662 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:20:14.985920 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:14.985889 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz"] Jul 22 13:20:14.989818 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:14.989787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:14.992229 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:14.992206 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ffc1de-serving-cert\"" Jul 22 13:20:15.010118 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.009994 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz"] Jul 22 13:20:15.088041 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.086854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58684b-445c-47c5-8997-5d73b3193813-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.088041 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.086916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wlx\" (UniqueName: \"kubernetes.io/projected/2e58684b-445c-47c5-8997-5d73b3193813-kube-api-access-p8wlx\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.088041 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.086976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.188239 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.188204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58684b-445c-47c5-8997-5d73b3193813-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.188239 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.188244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wlx\" (UniqueName: \"kubernetes.io/projected/2e58684b-445c-47c5-8997-5d73b3193813-kube-api-access-p8wlx\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.188492 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.188284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.188492 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:20:15.188390 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-serving-cert: secret "maas-api-e2e-shared-b-ffc1de-serving-cert" not found Jul 22 13:20:15.188492 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:20:15.188446 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls podName:2e58684b-445c-47c5-8997-5d73b3193813 nodeName:}" failed. No retries permitted until 2026-07-22 13:20:15.688430598 +0000 UTC m=+2369.493147472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls") pod "maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" (UID: "2e58684b-445c-47c5-8997-5d73b3193813") : secret "maas-api-e2e-shared-b-ffc1de-serving-cert" not found Jul 22 13:20:15.188873 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.188852 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58684b-445c-47c5-8997-5d73b3193813-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.199786 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.199750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wlx\" (UniqueName: \"kubernetes.io/projected/2e58684b-445c-47c5-8997-5d73b3193813-kube-api-access-p8wlx\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.692993 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.692961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.695675 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.695646 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls\") pod \"maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:15.909299 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:15.909243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:16.125774 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:16.125739 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz"] Jul 22 13:20:16.132834 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:20:16.132802 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e58684b_445c_47c5_8997_5d73b3193813.slice/crio-ce09ccacf924f15d039315c167287b8ab90fe77a462dd8f4c65bc08c0c39b098 WatchSource:0}: Error finding container ce09ccacf924f15d039315c167287b8ab90fe77a462dd8f4c65bc08c0c39b098: Status 404 returned error can't find the container with id ce09ccacf924f15d039315c167287b8ab90fe77a462dd8f4c65bc08c0c39b098 Jul 22 13:20:16.941380 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:16.941342 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" event={"ID":"2e58684b-445c-47c5-8997-5d73b3193813","Type":"ContainerStarted","Data":"8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0"} Jul 22 13:20:16.941758 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:16.941409 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" event={"ID":"2e58684b-445c-47c5-8997-5d73b3193813","Type":"ContainerStarted","Data":"ce09ccacf924f15d039315c167287b8ab90fe77a462dd8f4c65bc08c0c39b098"} Jul 22 13:20:16.941758 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:16.941529 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:16.960354 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:16.960312 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" podStartSLOduration=2.6473381160000002 podStartE2EDuration="2.960297326s" totalImagesPullingTime="312.95921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:20:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:20:16.957644095 +0000 UTC m=+2370.762360990" watchObservedRunningTime="2026-07-22 13:20:16.960297326 +0000 UTC m=+2370.765014221" Jul 22 13:20:21.787710 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:21.787354 2576 patch_prober.go:28] interesting pod/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": read tcp 10.132.0.2:42352->10.132.0.67:15021: read: connection reset by peer" start-of-body= Jul 22 13:20:21.787710 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:21.787428 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": read tcp 10.132.0.2:42352->10.132.0.67:15021: read: connection reset by peer" Jul 22 13:20:21.973700 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:21.973620 2576 generic.go:356] "Generic (PLEG): container finished" podID="065c950c-6f49-4283-b36c-63313aaafa19" containerID="d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853" exitCode=0 Jul 22 13:20:21.973844 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:21.973699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" event={"ID":"065c950c-6f49-4283-b36c-63313aaafa19","Type":"ContainerDied","Data":"d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853"} Jul 22 13:20:21.974154 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:21.974138 2576 scope.go:117] "RemoveContainer" containerID="d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853" Jul 22 13:20:21.975170 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:21.975137 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:21.975267 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:21.975248 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:21.975311 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:21.975299 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:22.955574 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:22.955547 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:20:22.981723 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:22.981688 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" event={"ID":"065c950c-6f49-4283-b36c-63313aaafa19","Type":"ContainerStarted","Data":"745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398"} Jul 22 13:20:23.406054 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:23.405936 2576 patch_prober.go:28] interesting pod/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 22 13:20:23.406333 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:23.406300 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 22 13:20:23.406702 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:23.406684 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:24.405349 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:24.405309 2576 patch_prober.go:28] interesting pod/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 22 13:20:24.406327 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:24.405915 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 22 13:20:24.796767 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:24.796731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:20:25.405495 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:25.405454 2576 patch_prober.go:28] interesting pod/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 22 13:20:25.405913 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:25.405518 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 22 13:20:26.406128 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:26.405661 2576 patch_prober.go:28] interesting pod/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 22 13:20:26.406128 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:26.405724 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 22 13:20:27.408884 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:27.408854 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:27.409301 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:27.409236 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:27.409850 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:27.409830 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:20:28.889891 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:28.889808 2576 patch_prober.go:28] interesting pod/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 22 13:20:28.889891 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:28.889864 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 22 13:20:29.018709 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:29.018677 2576 generic.go:356] "Generic (PLEG): container finished" podID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerID="a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d" exitCode=0 Jul 22 13:20:29.018876 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:29.018751 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" event={"ID":"50aa6168-7078-45a5-b040-8e5e22e7e82e","Type":"ContainerDied","Data":"a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d"} Jul 22 13:20:29.019252 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:29.019234 2576 scope.go:117] "RemoveContainer" containerID="a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d" Jul 22 13:20:29.020167 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:29.020143 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:29.020223 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:29.020206 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:29.020259 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:29.020241 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 22 13:20:30.028110 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:30.028073 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" event={"ID":"50aa6168-7078-45a5-b040-8e5e22e7e82e","Type":"ContainerStarted","Data":"931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd"} Jul 22 13:20:30.402762 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:30.402667 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:30.402762 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:30.402722 2576 patch_prober.go:28] interesting pod/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 22 13:20:30.403032 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:30.402783 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 22 13:20:31.402777 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:31.402741 2576 patch_prober.go:28] interesting pod/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 22 13:20:31.403490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:31.403457 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 22 13:20:32.402719 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:32.402680 2576 patch_prober.go:28] interesting pod/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 22 13:20:32.402921 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:32.402748 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 22 13:20:33.402365 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:33.402328 2576 patch_prober.go:28] interesting pod/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Jul 22 13:20:33.402670 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:33.402639 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Jul 22 13:20:34.407083 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:34.407055 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:34.407513 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:34.407494 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:34.408070 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:34.408048 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:20:46.194502 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:46.194469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:20:46.821982 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:46.821954 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:20:46.829346 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:46.829328 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:20:46.831753 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:46.831734 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:20:46.838673 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:20:46.838655 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:21:07.988261 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:21:07.988177 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:21:22.003602 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:21:22.003570 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:21:40.898230 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:21:40.898193 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:21:59.695035 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:21:59.694986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:22:14.194899 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:22:14.194863 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:22:34.209801 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:22:34.209771 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:22:54.426150 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:22:54.426106 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:23:14.925280 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:23:14.925234 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:23:29.499996 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:23:29.499960 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:24:02.700785 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:24:02.700752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:24:20.014048 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:24:20.013953 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:24:41.017322 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:24:41.017282 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:24:54.205332 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:24:54.205299 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:25:17.396706 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:25:17.396670 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:25:31.310835 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:25:31.310799 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:25:46.856750 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:25:46.856674 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:25:46.863688 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:25:46.863661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:25:46.868128 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:25:46.868112 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:25:46.874891 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:25:46.874876 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cdpvh_e8f8c309-00be-466a-94db-f8ce110a007f/ovn-acl-logging/0.log" Jul 22 13:25:47.112363 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:25:47.112295 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:26:06.096793 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:06.096752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:26:24.930710 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:24.930634 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg"] Jul 22 13:26:24.931151 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:24.930913 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" podUID="10a2dfe2-597b-4832-8782-20828a052a2e" containerName="maas-api" containerID="cri-o://a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37" gracePeriod=30 Jul 22 13:26:25.188514 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.188458 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:26:25.345920 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.345891 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a2dfe2-597b-4832-8782-20828a052a2e-openshift-service-ca\") pod \"10a2dfe2-597b-4832-8782-20828a052a2e\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " Jul 22 13:26:25.346103 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.345995 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8ln9\" (UniqueName: \"kubernetes.io/projected/10a2dfe2-597b-4832-8782-20828a052a2e-kube-api-access-v8ln9\") pod \"10a2dfe2-597b-4832-8782-20828a052a2e\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " Jul 22 13:26:25.346103 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.346043 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a2dfe2-597b-4832-8782-20828a052a2e-maas-api-tls\") pod \"10a2dfe2-597b-4832-8782-20828a052a2e\" (UID: \"10a2dfe2-597b-4832-8782-20828a052a2e\") " Jul 22 13:26:25.346288 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.346255 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a2dfe2-597b-4832-8782-20828a052a2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10a2dfe2-597b-4832-8782-20828a052a2e" (UID: "10a2dfe2-597b-4832-8782-20828a052a2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:26:25.348116 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.348091 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a2dfe2-597b-4832-8782-20828a052a2e-kube-api-access-v8ln9" (OuterVolumeSpecName: "kube-api-access-v8ln9") pod "10a2dfe2-597b-4832-8782-20828a052a2e" (UID: "10a2dfe2-597b-4832-8782-20828a052a2e"). InnerVolumeSpecName "kube-api-access-v8ln9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:25.348235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.348146 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a2dfe2-597b-4832-8782-20828a052a2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10a2dfe2-597b-4832-8782-20828a052a2e" (UID: "10a2dfe2-597b-4832-8782-20828a052a2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:26:25.446999 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.446945 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ln9\" (UniqueName: \"kubernetes.io/projected/10a2dfe2-597b-4832-8782-20828a052a2e-kube-api-access-v8ln9\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:25.446999 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.446968 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a2dfe2-597b-4832-8782-20828a052a2e-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:25.446999 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.446977 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a2dfe2-597b-4832-8782-20828a052a2e-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:25.940848 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.940815 2576 generic.go:356] "Generic (PLEG): container finished" podID="10a2dfe2-597b-4832-8782-20828a052a2e" containerID="a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37" exitCode=0 Jul 22 13:26:25.941235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.940856 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" Jul 22 13:26:25.941235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.940903 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" event={"ID":"10a2dfe2-597b-4832-8782-20828a052a2e","Type":"ContainerDied","Data":"a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37"} Jul 22 13:26:25.941235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.940957 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg" event={"ID":"10a2dfe2-597b-4832-8782-20828a052a2e","Type":"ContainerDied","Data":"6f77ab02a789ec30d3d0bd21ab0ee89e84c3b7a831daf10061c48d5ee9b53d99"} Jul 22 13:26:25.941235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.940977 2576 scope.go:117] "RemoveContainer" containerID="a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37" Jul 22 13:26:25.950819 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.950802 2576 scope.go:117] "RemoveContainer" containerID="a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37" Jul 22 13:26:25.951084 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:26:25.951066 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37\": container with ID starting with a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37 not found: ID does not exist" containerID="a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37" Jul 22 13:26:25.951150 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.951095 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37"} err="failed to get container status \"a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37\": rpc error: code = NotFound desc = could not find container \"a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37\": container with ID starting with a68afe751a52eebb1eca48b63df867b556bcf50871217619a244c90c95ed2d37 not found: ID does not exist" Jul 22 13:26:25.963110 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.963089 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg"] Jul 22 13:26:25.966602 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:25.966581 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24327-59c9cd6cdb-mhlwg"] Jul 22 13:26:26.713102 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:26.713072 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a2dfe2-597b-4832-8782-20828a052a2e" path="/var/lib/kubelet/pods/10a2dfe2-597b-4832-8782-20828a052a2e/volumes" Jul 22 13:26:28.099201 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:28.099165 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:26:36.560578 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:36.560547 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb"] Jul 22 13:26:36.560977 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:36.560754 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" containerID="cri-o://745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398" gracePeriod=30 Jul 22 13:26:41.814938 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.814913 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:26:41.886263 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886193 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-socket\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.886263 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886243 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c950c-6f49-4283-b36c-63313aaafa19-istio-podinfo\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.886470 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886284 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-certs\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.886470 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886302 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9l6w\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-kube-api-access-z9l6w\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.886470 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886323 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c950c-6f49-4283-b36c-63313aaafa19-istiod-ca-cert\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.886470 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-credential-socket\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.886470 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886444 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-istio-token\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.886720 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886473 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:41.886992 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-data\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.886992 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.886979 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/065c950c-6f49-4283-b36c-63313aaafa19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:26:41.887168 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.887055 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-envoy\") pod \"065c950c-6f49-4283-b36c-63313aaafa19\" (UID: \"065c950c-6f49-4283-b36c-63313aaafa19\") " Jul 22 13:26:41.887339 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.887316 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:41.887523 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.887454 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:41.887894 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.887842 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-certs\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.887894 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.887866 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/065c950c-6f49-4283-b36c-63313aaafa19-istiod-ca-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.887894 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.887879 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-credential-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.887894 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.887892 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-workload-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.888241 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.888004 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-data" (OuterVolumeSpecName: "istio-data") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:41.890982 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.890958 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:41.895839 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.895796 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-kube-api-access-z9l6w" (OuterVolumeSpecName: "kube-api-access-z9l6w") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "kube-api-access-z9l6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:41.895839 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.895823 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-istio-token" (OuterVolumeSpecName: "istio-token") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:41.895992 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.895930 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/065c950c-6f49-4283-b36c-63313aaafa19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "065c950c-6f49-4283-b36c-63313aaafa19" (UID: "065c950c-6f49-4283-b36c-63313aaafa19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:26:41.988392 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.988369 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/065c950c-6f49-4283-b36c-63313aaafa19-istio-podinfo\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.988392 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.988393 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9l6w\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-kube-api-access-z9l6w\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.988525 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.988404 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/065c950c-6f49-4283-b36c-63313aaafa19-istio-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.988525 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.988412 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-data\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:41.988525 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:41.988419 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/065c950c-6f49-4283-b36c-63313aaafa19-istio-envoy\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:42.030761 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.030734 2576 generic.go:356] "Generic (PLEG): container finished" podID="065c950c-6f49-4283-b36c-63313aaafa19" containerID="745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398" exitCode=0 Jul 22 13:26:42.030876 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.030792 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" Jul 22 13:26:42.030876 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.030825 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" event={"ID":"065c950c-6f49-4283-b36c-63313aaafa19","Type":"ContainerDied","Data":"745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398"} Jul 22 13:26:42.030996 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.030917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb" event={"ID":"065c950c-6f49-4283-b36c-63313aaafa19","Type":"ContainerDied","Data":"6943a2cda2de8ff051938893905de8157af263f968616e55e6f64fc4a5ede552"} Jul 22 13:26:42.030996 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.030950 2576 scope.go:117] "RemoveContainer" containerID="745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398" Jul 22 13:26:42.040689 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.040667 2576 scope.go:117] "RemoveContainer" containerID="d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853" Jul 22 13:26:42.048443 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.048427 2576 scope.go:117] "RemoveContainer" containerID="745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398" Jul 22 13:26:42.048666 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:26:42.048647 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398\": container with ID starting with 745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398 not found: ID does not exist" containerID="745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398" Jul 22 13:26:42.048742 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.048677 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398"} err="failed to get container status \"745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398\": rpc error: code = NotFound desc = could not find container \"745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398\": container with ID starting with 745690ffc52dc1a97dfaebce6eae229aaedc72c644815a460222749cf97b2398 not found: ID does not exist" Jul 22 13:26:42.048742 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.048699 2576 scope.go:117] "RemoveContainer" containerID="d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853" Jul 22 13:26:42.048912 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:26:42.048894 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853\": container with ID starting with d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853 not found: ID does not exist" containerID="d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853" Jul 22 13:26:42.048953 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.048918 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853"} err="failed to get container status \"d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853\": rpc error: code = NotFound desc = could not find container \"d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853\": container with ID starting with d4b7a1b00d286665c9fb343785876d05865d9f2ec7f5384e1fa4d0ce61bfb853 not found: ID does not exist" Jul 22 13:26:42.053143 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.053123 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb"] Jul 22 13:26:42.059369 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.059352 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d24327-openshift-default-56b9ff5dff-whplb"] Jul 22 13:26:42.712723 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:42.712693 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065c950c-6f49-4283-b36c-63313aaafa19" path="/var/lib/kubelet/pods/065c950c-6f49-4283-b36c-63313aaafa19/volumes" Jul 22 13:26:47.465611 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.465582 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz"] Jul 22 13:26:47.466101 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.465841 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" podUID="2e58684b-445c-47c5-8997-5d73b3193813" containerName="maas-api" containerID="cri-o://8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0" gracePeriod=30 Jul 22 13:26:47.713348 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.713320 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:26:47.837787 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.837713 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls\") pod \"2e58684b-445c-47c5-8997-5d73b3193813\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " Jul 22 13:26:47.837787 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.837768 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8wlx\" (UniqueName: \"kubernetes.io/projected/2e58684b-445c-47c5-8997-5d73b3193813-kube-api-access-p8wlx\") pod \"2e58684b-445c-47c5-8997-5d73b3193813\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " Jul 22 13:26:47.837965 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.837832 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58684b-445c-47c5-8997-5d73b3193813-openshift-service-ca\") pod \"2e58684b-445c-47c5-8997-5d73b3193813\" (UID: \"2e58684b-445c-47c5-8997-5d73b3193813\") " Jul 22 13:26:47.838221 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.838194 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e58684b-445c-47c5-8997-5d73b3193813-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e58684b-445c-47c5-8997-5d73b3193813" (UID: "2e58684b-445c-47c5-8997-5d73b3193813"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:26:47.839925 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.839903 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e58684b-445c-47c5-8997-5d73b3193813-kube-api-access-p8wlx" (OuterVolumeSpecName: "kube-api-access-p8wlx") pod "2e58684b-445c-47c5-8997-5d73b3193813" (UID: "2e58684b-445c-47c5-8997-5d73b3193813"). InnerVolumeSpecName "kube-api-access-p8wlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:47.840033 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.839981 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e58684b-445c-47c5-8997-5d73b3193813" (UID: "2e58684b-445c-47c5-8997-5d73b3193813"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:26:47.939071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.939048 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e58684b-445c-47c5-8997-5d73b3193813-maas-api-tls\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:47.939071 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.939068 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8wlx\" (UniqueName: \"kubernetes.io/projected/2e58684b-445c-47c5-8997-5d73b3193813-kube-api-access-p8wlx\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:47.939190 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:47.939097 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e58684b-445c-47c5-8997-5d73b3193813-openshift-service-ca\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:26:48.063562 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.063537 2576 generic.go:356] "Generic (PLEG): container finished" podID="2e58684b-445c-47c5-8997-5d73b3193813" containerID="8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0" exitCode=0 Jul 22 13:26:48.063681 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.063582 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" Jul 22 13:26:48.063681 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.063603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" event={"ID":"2e58684b-445c-47c5-8997-5d73b3193813","Type":"ContainerDied","Data":"8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0"} Jul 22 13:26:48.063681 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.063645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz" event={"ID":"2e58684b-445c-47c5-8997-5d73b3193813","Type":"ContainerDied","Data":"ce09ccacf924f15d039315c167287b8ab90fe77a462dd8f4c65bc08c0c39b098"} Jul 22 13:26:48.063681 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.063665 2576 scope.go:117] "RemoveContainer" containerID="8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0" Jul 22 13:26:48.072490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.072473 2576 scope.go:117] "RemoveContainer" containerID="8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0" Jul 22 13:26:48.072737 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:26:48.072719 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0\": container with ID starting with 8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0 not found: ID does not exist" containerID="8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0" Jul 22 13:26:48.072800 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.072748 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0"} err="failed to get container status \"8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0\": rpc error: code = NotFound desc = could not find container \"8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0\": container with ID starting with 8cd2a69b8fee32bf05855c6eb90796eef61e98be5b1d744ee26ba4197b443ee0 not found: ID does not exist" Jul 22 13:26:48.085144 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.085122 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz"] Jul 22 13:26:48.089113 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.089069 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ffc1de-7689c8b546-tg2cz"] Jul 22 13:26:48.714319 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:48.714288 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e58684b-445c-47c5-8997-5d73b3193813" path="/var/lib/kubelet/pods/2e58684b-445c-47c5-8997-5d73b3193813/volumes" Jul 22 13:26:50.197459 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:50.197429 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-k57ld"] Jul 22 13:26:58.865235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:58.865204 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd"] Jul 22 13:26:58.865692 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:26:58.865411 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" containerID="cri-o://931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd" gracePeriod=30 Jul 22 13:27:04.120096 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.120076 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:27:04.146483 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.146418 2576 generic.go:356] "Generic (PLEG): container finished" podID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerID="931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd" exitCode=0 Jul 22 13:27:04.146605 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.146476 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" Jul 22 13:27:04.146605 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.146484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" event={"ID":"50aa6168-7078-45a5-b040-8e5e22e7e82e","Type":"ContainerDied","Data":"931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd"} Jul 22 13:27:04.146605 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.146535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd" event={"ID":"50aa6168-7078-45a5-b040-8e5e22e7e82e","Type":"ContainerDied","Data":"5dc1527463acf19f46009858795d590bac4e4aa65681fd775f0e720caca33f3b"} Jul 22 13:27:04.146605 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.146565 2576 scope.go:117] "RemoveContainer" containerID="931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd" Jul 22 13:27:04.155342 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.155321 2576 scope.go:117] "RemoveContainer" containerID="a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d" Jul 22 13:27:04.163541 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.163508 2576 scope.go:117] "RemoveContainer" containerID="931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd" Jul 22 13:27:04.163762 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:27:04.163741 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd\": container with ID starting with 931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd not found: ID does not exist" containerID="931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd" Jul 22 13:27:04.163832 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.163768 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd"} err="failed to get container status \"931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd\": rpc error: code = NotFound desc = could not find container \"931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd\": container with ID starting with 931fbd0314279cebc5a5758f989a9ff4f3d0d1a6de71d9414f653f1a0c5305bd not found: ID does not exist" Jul 22 13:27:04.163832 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.163790 2576 scope.go:117] "RemoveContainer" containerID="a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d" Jul 22 13:27:04.164092 ip-10-0-137-88 kubenswrapper[2576]: E0722 13:27:04.164073 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d\": container with ID starting with a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d not found: ID does not exist" containerID="a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d" Jul 22 13:27:04.164181 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.164107 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d"} err="failed to get container status \"a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d\": rpc error: code = NotFound desc = could not find container \"a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d\": container with ID starting with a7ef09ea8b00ed19232311e1b2860d1c41e64152be30e8f628c3e43a27d11d5d not found: ID does not exist" Jul 22 13:27:04.262470 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262446 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-podinfo\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.262595 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262493 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-envoy\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.262595 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262510 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-certs\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.262595 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262551 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6168-7078-45a5-b040-8e5e22e7e82e-istiod-ca-cert\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.262756 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262610 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqnl2\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-kube-api-access-mqnl2\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.262756 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262631 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-data\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.262756 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262651 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-credential-socket\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.262756 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262700 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-token\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.262756 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262727 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-socket\") pod \"50aa6168-7078-45a5-b040-8e5e22e7e82e\" (UID: \"50aa6168-7078-45a5-b040-8e5e22e7e82e\") " Jul 22 13:27:04.263043 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262856 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:27:04.263043 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262961 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-certs\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.263043 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.262959 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50aa6168-7078-45a5-b040-8e5e22e7e82e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:27:04.263361 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.263097 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:27:04.263449 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.263425 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:27:04.263512 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.263457 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-data" (OuterVolumeSpecName: "istio-data") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:27:04.264690 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.264668 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:27:04.264690 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.264674 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 22 13:27:04.264822 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.264737 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-kube-api-access-mqnl2" (OuterVolumeSpecName: "kube-api-access-mqnl2") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "kube-api-access-mqnl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:27:04.265238 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.265224 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-token" (OuterVolumeSpecName: "istio-token") pod "50aa6168-7078-45a5-b040-8e5e22e7e82e" (UID: "50aa6168-7078-45a5-b040-8e5e22e7e82e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:27:04.364235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.364205 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50aa6168-7078-45a5-b040-8e5e22e7e82e-istiod-ca-cert\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.364235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.364235 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqnl2\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-kube-api-access-mqnl2\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.364356 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.364244 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-data\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.364356 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.364252 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-credential-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.364356 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.364260 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-token\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.364356 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.364268 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-workload-socket\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.364356 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.364276 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-podinfo\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.364356 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.364283 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50aa6168-7078-45a5-b040-8e5e22e7e82e-istio-envoy\") on node \"ip-10-0-137-88.ec2.internal\" DevicePath \"\"" Jul 22 13:27:04.469101 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.469078 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd"] Jul 22 13:27:04.475517 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.475495 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-ffc1de-openshift-default-6cb8c9465f-kwvtd"] Jul 22 13:27:04.713665 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:04.713607 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" path="/var/lib/kubelet/pods/50aa6168-7078-45a5-b040-8e5e22e7e82e/volumes" Jul 22 13:27:05.306468 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:05.306440 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vz6lc_f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe/manager/0.log" Jul 22 13:27:05.423844 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:05.423818 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-66ssc_380a3bc1-3f05-496d-834c-a7df7a8104ee/manager/0.log" Jul 22 13:27:05.553313 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:05.553158 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c5545ff5f-wfzpv_92e2671d-18e7-49dd-9dee-faf7fbcd6888/manager/0.log" Jul 22 13:27:05.666692 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:05.666673 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-svf46_07f9501f-bf05-47e3-ae81-c99b2a69e25c/server/0.log" Jul 22 13:27:05.953316 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:05.953247 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-8ztbs_55374e50-ebda-4d7d-ad74-953f5ba81c9c/copy-manifests/0.log" Jul 22 13:27:05.966186 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:05.966166 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-8ztbs_55374e50-ebda-4d7d-ad74-953f5ba81c9c/manager/0.log" Jul 22 13:27:06.586497 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:06.586474 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6dfb8748db-pv8zd_73983dd7-6c72-4b48-ae6d-9c4a27f737d2/maas-api/0.log" Jul 22 13:27:07.764395 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:07.764322 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn_ea53e9ad-7714-494f-b698-e164671c6ab7/util/0.log" Jul 22 13:27:07.774154 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:07.774131 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn_ea53e9ad-7714-494f-b698-e164671c6ab7/pull/0.log" Jul 22 13:27:07.781895 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:07.781870 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn_ea53e9ad-7714-494f-b698-e164671c6ab7/extract/0.log" Jul 22 13:27:07.895314 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:07.895293 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv_3f31666c-b4a9-4e7e-b2df-29bf5be874aa/extract/0.log" Jul 22 13:27:07.902265 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:07.902247 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv_3f31666c-b4a9-4e7e-b2df-29bf5be874aa/util/0.log" Jul 22 13:27:07.910004 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:07.909982 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv_3f31666c-b4a9-4e7e-b2df-29bf5be874aa/pull/0.log" Jul 22 13:27:08.031747 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:08.031684 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c_aa7d0a0a-964f-4ede-ab83-83abdb9a9525/util/0.log" Jul 22 13:27:08.040006 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:08.039989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c_aa7d0a0a-964f-4ede-ab83-83abdb9a9525/pull/0.log" Jul 22 13:27:08.047848 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:08.047834 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c_aa7d0a0a-964f-4ede-ab83-83abdb9a9525/extract/0.log" Jul 22 13:27:08.168114 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:08.168081 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf_0d2bfb32-d761-4047-8be9-86e52e4ec9fb/extract/0.log" Jul 22 13:27:08.176736 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:08.176720 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf_0d2bfb32-d761-4047-8be9-86e52e4ec9fb/util/0.log" Jul 22 13:27:08.184737 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:08.184723 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf_0d2bfb32-d761-4047-8be9-86e52e4ec9fb/pull/0.log" Jul 22 13:27:09.088537 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:09.088501 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-k57ld_300abdfe-4403-45c2-b7e1-2a3316aa96d6/limitador/0.log" Jul 22 13:27:09.735121 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:09.735097 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vqg5f_87215607-2efc-4c09-82f1-d9ef64a78026/discovery/0.log" Jul 22 13:27:09.859155 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:09.859134 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c9b95cb4-6pb8h_17c4a082-5d5a-4e10-aec1-5e2f88fa12fb/kube-auth-proxy/0.log" Jul 22 13:27:10.241882 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:10.241855 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hggtd_6154c67f-3cfd-471b-b1c4-847e905ab587/payload-pre-processing/0.log" Jul 22 13:27:10.361400 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:10.361379 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-wgztj_aeff253e-7449-44f6-896c-c3172c83cc8d/payload-processing/0.log" Jul 22 13:27:10.478183 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:10.478159 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c964bf7f6-tqffj_5e7a2923-60f8-434a-9cc3-2cc73825ddcf/router/0.log" Jul 22 13:27:11.103817 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:11.103795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nntfz_4fcd1130-cef4-49c6-ba35-36728bd2bf1a/main/0.log" Jul 22 13:27:19.695431 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:19.695392 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4hvsk_e10dae84-5ee9-4c3c-a86c-608de7b8504e/global-pull-secret-syncer/0.log" Jul 22 13:27:19.868967 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:19.868937 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l9jtw_a1347662-9683-4412-9e63-61017d9d3be3/konnectivity-agent/0.log" Jul 22 13:27:19.970911 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:19.970853 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-88.ec2.internal_349801908a8762aee344d69fa297f1e3/haproxy/0.log" Jul 22 13:27:24.791027 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.790990 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn_ea53e9ad-7714-494f-b698-e164671c6ab7/extract/0.log" Jul 22 13:27:24.815231 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.815206 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn_ea53e9ad-7714-494f-b698-e164671c6ab7/util/0.log" Jul 22 13:27:24.839794 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.839772 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ndwrn_ea53e9ad-7714-494f-b698-e164671c6ab7/pull/0.log" Jul 22 13:27:24.879432 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.879406 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv_3f31666c-b4a9-4e7e-b2df-29bf5be874aa/extract/0.log" Jul 22 13:27:24.903102 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.903084 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv_3f31666c-b4a9-4e7e-b2df-29bf5be874aa/util/0.log" Jul 22 13:27:24.925043 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.925006 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0z4rlv_3f31666c-b4a9-4e7e-b2df-29bf5be874aa/pull/0.log" Jul 22 13:27:24.952319 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.952303 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c_aa7d0a0a-964f-4ede-ab83-83abdb9a9525/extract/0.log" Jul 22 13:27:24.977844 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.977827 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c_aa7d0a0a-964f-4ede-ab83-83abdb9a9525/util/0.log" Jul 22 13:27:24.998876 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:24.998857 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739pw6c_aa7d0a0a-964f-4ede-ab83-83abdb9a9525/pull/0.log" Jul 22 13:27:25.024729 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:25.024707 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf_0d2bfb32-d761-4047-8be9-86e52e4ec9fb/extract/0.log" Jul 22 13:27:25.048939 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:25.048881 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf_0d2bfb32-d761-4047-8be9-86e52e4ec9fb/util/0.log" Jul 22 13:27:25.068988 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:25.068971 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cbjsf_0d2bfb32-d761-4047-8be9-86e52e4ec9fb/pull/0.log" Jul 22 13:27:25.846051 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:25.846027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-k57ld_300abdfe-4403-45c2-b7e1-2a3316aa96d6/limitador/0.log" Jul 22 13:27:27.667806 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:27.667778 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-8fq6j_f68990c1-be9f-4a0a-9165-cb17dcfa2ec3/cluster-monitoring-operator/0.log" Jul 22 13:27:27.908316 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:27.908291 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f7d5q_d32b678c-ced0-4f12-ad48-02f6dde24b37/node-exporter/0.log" Jul 22 13:27:27.931777 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:27.931719 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f7d5q_d32b678c-ced0-4f12-ad48-02f6dde24b37/kube-rbac-proxy/0.log" Jul 22 13:27:27.953447 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:27.953430 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f7d5q_d32b678c-ced0-4f12-ad48-02f6dde24b37/init-textfile/0.log" Jul 22 13:27:29.511130 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511100 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb"] Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511422 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e58684b-445c-47c5-8997-5d73b3193813" containerName="maas-api" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511431 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e58684b-445c-47c5-8997-5d73b3193813" containerName="maas-api" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511442 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511448 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511455 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511462 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511468 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511473 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511487 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a2dfe2-597b-4832-8782-20828a052a2e" containerName="maas-api" Jul 22 13:27:29.511490 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511492 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a2dfe2-597b-4832-8782-20828a052a2e" containerName="maas-api" Jul 22 13:27:29.511794 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511554 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50aa6168-7078-45a5-b040-8e5e22e7e82e" containerName="istio-proxy" Jul 22 13:27:29.511794 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511562 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" Jul 22 13:27:29.511794 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511567 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="065c950c-6f49-4283-b36c-63313aaafa19" containerName="istio-proxy" Jul 22 13:27:29.511794 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511571 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e58684b-445c-47c5-8997-5d73b3193813" containerName="maas-api" Jul 22 13:27:29.511794 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.511577 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a2dfe2-597b-4832-8782-20828a052a2e" containerName="maas-api" Jul 22 13:27:29.514619 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.514600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.517223 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.517204 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g2jzv\"/\"openshift-service-ca.crt\"" Jul 22 13:27:29.517924 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.517908 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g2jzv\"/\"default-dockercfg-5xtwz\"" Jul 22 13:27:29.517980 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.517949 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g2jzv\"/\"kube-root-ca.crt\"" Jul 22 13:27:29.524336 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.524314 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb"] Jul 22 13:27:29.654828 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.654800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-podres\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.655034 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.654835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-sys\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.655034 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.654940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-proc\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.655034 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.654971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-lib-modules\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.655034 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.654998 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dn5r\" (UniqueName: \"kubernetes.io/projected/4560bad6-96b7-4589-8e35-5f882b037d45-kube-api-access-9dn5r\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.755862 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.755834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-proc\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.755862 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.755865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-lib-modules\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.756123 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.755882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dn5r\" (UniqueName: \"kubernetes.io/projected/4560bad6-96b7-4589-8e35-5f882b037d45-kube-api-access-9dn5r\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.756123 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.755925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-podres\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.756123 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.755951 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-proc\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.756123 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.756050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-podres\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.756123 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.756055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-lib-modules\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.756123 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.756054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-sys\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.756123 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.756092 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4560bad6-96b7-4589-8e35-5f882b037d45-sys\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.764591 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.764534 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dn5r\" (UniqueName: \"kubernetes.io/projected/4560bad6-96b7-4589-8e35-5f882b037d45-kube-api-access-9dn5r\") pod \"perf-node-gather-daemonset-kzzlb\" (UID: \"4560bad6-96b7-4589-8e35-5f882b037d45\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.825395 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.825373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:29.955817 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:29.955789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb"] Jul 22 13:27:29.956897 ip-10-0-137-88 kubenswrapper[2576]: W0722 13:27:29.956872 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4560bad6_96b7_4589_8e35_5f882b037d45.slice/crio-a13898643acd28decfe37fd0f9ce73025705f8c79b56a6a48d4e835c4fb41e80 WatchSource:0}: Error finding container a13898643acd28decfe37fd0f9ce73025705f8c79b56a6a48d4e835c4fb41e80: Status 404 returned error can't find the container with id a13898643acd28decfe37fd0f9ce73025705f8c79b56a6a48d4e835c4fb41e80 Jul 22 13:27:30.279048 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:30.279021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" event={"ID":"4560bad6-96b7-4589-8e35-5f882b037d45","Type":"ContainerStarted","Data":"3983dad4c162e68efe95e48bedadcb4d65f8e5fbc6b83f271903d957c2496cf5"} Jul 22 13:27:30.279207 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:30.279069 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" event={"ID":"4560bad6-96b7-4589-8e35-5f882b037d45","Type":"ContainerStarted","Data":"a13898643acd28decfe37fd0f9ce73025705f8c79b56a6a48d4e835c4fb41e80"} Jul 22 13:27:30.279207 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:30.279135 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:30.298054 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:30.297998 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" podStartSLOduration=1.297986083 podStartE2EDuration="1.297986083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 13:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:27:30.296247557 +0000 UTC m=+2804.100964455" watchObservedRunningTime="2026-07-22 13:27:30.297986083 +0000 UTC m=+2804.102702979" Jul 22 13:27:30.487546 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:30.487522 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/1.log" Jul 22 13:27:30.492644 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:30.492625 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-rck2w_0550d49e-243e-437e-b4a6-5badb236b4fe/console-operator/2.log" Jul 22 13:27:31.616258 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:31.616231 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-jwg8w_e61b8b11-07b7-413a-87e5-525e3a5f230d/volume-data-source-validator/0.log" Jul 22 13:27:32.702246 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:32.702217 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qkvwk_72cd096f-6bab-4c50-9088-186c56d2a701/dns/0.log" Jul 22 13:27:32.725078 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:32.725055 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qkvwk_72cd096f-6bab-4c50-9088-186c56d2a701/kube-rbac-proxy/0.log" Jul 22 13:27:32.833206 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:32.833183 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qh6q7_713f7499-9b3a-4f9c-b6a8-8682fb8f507d/dns-node-resolver/0.log" Jul 22 13:27:33.449806 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:33.449774 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7kfr8_cdab67a7-9489-4d5c-ad85-098e8a5c7c7d/node-ca/0.log" Jul 22 13:27:34.604831 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:34.604804 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vqg5f_87215607-2efc-4c09-82f1-d9ef64a78026/discovery/0.log" Jul 22 13:27:34.626914 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:34.626892 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c9b95cb4-6pb8h_17c4a082-5d5a-4e10-aec1-5e2f88fa12fb/kube-auth-proxy/0.log" Jul 22 13:27:34.752948 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:34.752909 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hggtd_6154c67f-3cfd-471b-b1c4-847e905ab587/payload-pre-processing/0.log" Jul 22 13:27:34.784410 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:34.784363 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-wgztj_aeff253e-7449-44f6-896c-c3172c83cc8d/payload-processing/0.log" Jul 22 13:27:34.806822 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:34.806792 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c964bf7f6-tqffj_5e7a2923-60f8-434a-9cc3-2cc73825ddcf/router/0.log" Jul 22 13:27:35.540765 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:35.540738 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mpmlz_c729e819-258c-496d-befc-dca46019ce2c/serve-healthcheck-canary/0.log" Jul 22 13:27:35.998868 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:35.998840 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-8d228_475631b0-ff58-405b-bdca-7228773d131d/insights-operator/0.log" Jul 22 13:27:36.000004 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:35.999983 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-8d228_475631b0-ff58-405b-bdca-7228773d131d/insights-operator/1.log" Jul 22 13:27:36.158142 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:36.158118 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rb9sd_ef7f534d-ea89-4086-8ad1-d7a303cc8ec8/kube-rbac-proxy/0.log" Jul 22 13:27:36.177685 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:36.177661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rb9sd_ef7f534d-ea89-4086-8ad1-d7a303cc8ec8/exporter/0.log" Jul 22 13:27:36.210553 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:36.210532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rb9sd_ef7f534d-ea89-4086-8ad1-d7a303cc8ec8/extractor/0.log" Jul 22 13:27:36.294134 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:36.294084 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-kzzlb" Jul 22 13:27:38.704235 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:38.704186 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vz6lc_f59f1c5c-bf40-4b22-9cf6-fd36bb0e0abe/manager/0.log" Jul 22 13:27:38.869638 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:38.869616 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-66ssc_380a3bc1-3f05-496d-834c-a7df7a8104ee/manager/0.log" Jul 22 13:27:38.987289 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:38.987221 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6c5545ff5f-wfzpv_92e2671d-18e7-49dd-9dee-faf7fbcd6888/manager/0.log" Jul 22 13:27:39.013543 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:39.013527 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-svf46_07f9501f-bf05-47e3-ae81-c99b2a69e25c/server/0.log" Jul 22 13:27:39.178827 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:39.178809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-8ztbs_55374e50-ebda-4d7d-ad74-953f5ba81c9c/manager/0.log" Jul 22 13:27:39.198034 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:39.197991 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-8ztbs_55374e50-ebda-4d7d-ad74-953f5ba81c9c/copy-manifests/0.log" Jul 22 13:27:40.378546 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:40.378514 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vqg5f_87215607-2efc-4c09-82f1-d9ef64a78026/discovery/0.log" Jul 22 13:27:40.388438 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:40.388418 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c9b95cb4-6pb8h_17c4a082-5d5a-4e10-aec1-5e2f88fa12fb/kube-auth-proxy/0.log" Jul 22 13:27:40.481971 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:40.481945 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hggtd_6154c67f-3cfd-471b-b1c4-847e905ab587/payload-pre-processing/0.log" Jul 22 13:27:40.499819 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:40.499793 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-wgztj_aeff253e-7449-44f6-896c-c3172c83cc8d/payload-processing/0.log" Jul 22 13:27:40.511924 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:40.511907 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c964bf7f6-tqffj_5e7a2923-60f8-434a-9cc3-2cc73825ddcf/router/0.log" Jul 22 13:27:41.845761 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:41.845733 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-vqg5f_87215607-2efc-4c09-82f1-d9ef64a78026/discovery/0.log" Jul 22 13:27:41.858255 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:41.858231 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76c9b95cb4-6pb8h_17c4a082-5d5a-4e10-aec1-5e2f88fa12fb/kube-auth-proxy/0.log" Jul 22 13:27:41.957567 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:41.957543 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-hggtd_6154c67f-3cfd-471b-b1c4-847e905ab587/payload-pre-processing/0.log" Jul 22 13:27:41.976309 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:41.976283 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-wgztj_aeff253e-7449-44f6-896c-c3172c83cc8d/payload-processing/0.log" Jul 22 13:27:41.990731 ip-10-0-137-88 kubenswrapper[2576]: I0722 13:27:41.990711 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c964bf7f6-tqffj_5e7a2923-60f8-434a-9cc3-2cc73825ddcf/router/0.log"