Jul 09 09:52:18.215698 ip-10-0-140-69 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 09:52:18.215708 ip-10-0-140-69 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 09:52:18.215718 ip-10-0-140-69 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 09:52:18.216055 ip-10-0-140-69 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 09:52:28.288834 ip-10-0-140-69 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 09:52:28.288853 ip-10-0-140-69 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7847954837ec417096dc8979e87324cd -- Jul 09 09:54:04.556407 ip-10-0-140-69 systemd[1]: Starting Kubernetes Kubelet... Jul 09 09:54:04.938187 ip-10-0-140-69 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 09:54:04.938187 ip-10-0-140-69 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 09:54:04.938187 ip-10-0-140-69 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 09:54:04.938187 ip-10-0-140-69 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 09:54:04.938187 ip-10-0-140-69 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 09:54:04.939962 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.939880 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 09:54:04.944313 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944298 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:54:04.944313 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944314 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944317 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944321 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944324 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944327 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944330 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944334 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944336 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944344 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944349 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944353 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944356 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944359 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944361 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944364 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944367 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944370 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944373 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944376 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944379 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:54:04.944376 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944382 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944387 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944392 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944394 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944397 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944400 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944403 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944405 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944408 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944410 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944413 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944416 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944418 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944421 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944423 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944426 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944428 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944431 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944434 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944436 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:54:04.944844 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944439 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944442 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944446 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944450 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944453 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944456 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944459 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944461 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944464 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944466 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944469 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944471 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944474 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944477 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944480 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944482 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944485 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944487 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944489 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:54:04.945371 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944492 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944495 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944497 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944499 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944502 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944504 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944507 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944509 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944512 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944514 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944517 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944520 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944522 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944525 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944528 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944530 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944532 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944535 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944537 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:54:04.945823 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944540 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944543 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944545 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944548 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944550 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944552 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944556 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944558 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944561 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944564 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944566 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944569 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944571 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944574 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944576 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944579 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944581 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944584 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944586 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.944588 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:54:04.946304 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945271 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945278 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945281 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945284 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945287 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945292 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945296 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945299 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945302 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945305 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945308 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945312 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945316 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945319 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945321 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945324 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945327 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945330 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945333 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:54:04.946810 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945335 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945338 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945341 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945344 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945346 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945349 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945351 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945354 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945357 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945360 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945362 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945365 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945367 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945370 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945373 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945375 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945378 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945381 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945383 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945387 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:54:04.947292 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945390 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945392 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945395 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945398 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945400 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945403 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945405 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945408 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945410 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945413 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945415 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945418 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945420 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945423 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945426 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945428 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945431 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945433 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945435 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945438 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:54:04.947808 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945443 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945447 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945449 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945452 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945454 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945457 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945460 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945463 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945466 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945468 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945471 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945474 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945477 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945479 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945482 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945485 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945487 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945490 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945492 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945495 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:54:04.948336 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945498 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945500 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945503 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945505 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945508 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945511 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945513 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945516 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945518 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945521 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945524 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945526 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945529 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945531 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945534 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945536 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945538 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945541 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945543 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945546 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:54:04.948820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945902 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945906 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945909 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945912 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945915 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945918 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945921 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945923 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945928 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945931 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945934 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945937 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945940 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945943 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945946 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945949 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945953 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945955 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945958 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:54:04.949321 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945961 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945963 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945966 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945969 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945971 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945974 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945976 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945978 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945981 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945983 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945986 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945988 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945991 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945993 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945996 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.945998 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946001 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946004 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946007 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:54:04.949774 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946009 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946012 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946014 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946016 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946019 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946021 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946024 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946026 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946028 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946031 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946034 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946038 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946040 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946043 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946045 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946047 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946050 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946053 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946056 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946059 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:54:04.950254 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946061 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946064 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946066 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946069 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946071 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946074 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946076 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946078 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946081 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946084 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946086 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946089 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946094 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946097 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946100 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946102 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946105 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946108 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946110 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946113 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:54:04.950742 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946115 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946118 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946135 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946138 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946140 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946143 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946145 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946148 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946150 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946153 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946155 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946158 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946161 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946163 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946165 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946168 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946171 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946173 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946175 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946178 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946181 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:54:04.951241 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946247 2571 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946254 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946260 2571 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946264 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946269 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946272 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946276 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946281 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946284 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946287 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946290 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946293 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946297 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946299 2571 flags.go:64] FLAG: --cgroup-root="" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946302 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946305 2571 flags.go:64] FLAG: --client-ca-file="" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946308 2571 flags.go:64] FLAG: --cloud-provider="external" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946312 2571 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946317 2571 flags.go:64] FLAG: --cluster-domain="" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946320 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946323 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946326 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946329 2571 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946333 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 09:54:04.951737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946336 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946338 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946342 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946345 2571 flags.go:64] FLAG: --contention-profiling="false" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946347 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946350 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946353 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946356 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946360 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946363 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946366 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946369 2571 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946372 2571 flags.go:64] FLAG: --enable-server="true" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946383 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946387 2571 flags.go:64] FLAG: --event-burst="100" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946390 2571 flags.go:64] FLAG: --event-qps="50" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946393 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946396 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946399 2571 flags.go:64] FLAG: --eviction-hard="" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946407 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946410 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946413 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946416 2571 flags.go:64] FLAG: --eviction-soft="" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946419 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946422 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 09:54:04.952352 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946425 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946428 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946431 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946434 2571 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946437 2571 flags.go:64] FLAG: --feature-gates="" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946440 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946443 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946447 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946450 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946453 2571 flags.go:64] FLAG: --healthz-port="10248" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946456 2571 flags.go:64] FLAG: --help="false" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946459 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-140-69.ec2.internal" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946462 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946465 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946468 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946471 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946475 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946477 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946480 2571 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946483 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946486 2571 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946490 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946493 2571 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946496 2571 flags.go:64] FLAG: --kube-reserved="" Jul 09 09:54:04.953027 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946499 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946504 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946507 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946510 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946513 2571 flags.go:64] FLAG: --lock-file="" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946516 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946518 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946522 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946528 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946531 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946534 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946537 2571 flags.go:64] FLAG: --logging-format="text" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946540 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946544 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946547 2571 flags.go:64] FLAG: --manifest-url="" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946550 2571 flags.go:64] FLAG: --manifest-url-header="" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946554 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946557 2571 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946561 2571 flags.go:64] FLAG: --max-pods="110" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946564 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946566 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946569 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946572 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946575 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946578 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 09:54:04.953634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946581 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946588 2571 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946591 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946594 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946596 2571 flags.go:64] FLAG: --pod-cidr="" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946600 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946604 2571 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946607 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946610 2571 flags.go:64] FLAG: --pods-per-core="0" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946617 2571 flags.go:64] FLAG: --port="10250" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946619 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946622 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07c37ebdf907ae146" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946625 2571 flags.go:64] FLAG: --qos-reserved="" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946628 2571 flags.go:64] FLAG: --read-only-port="10255" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946631 2571 flags.go:64] FLAG: --register-node="true" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946639 2571 flags.go:64] FLAG: --register-with-taints="" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946642 2571 flags.go:64] FLAG: --registry-burst="10" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946645 2571 flags.go:64] FLAG: --registry-qps="5" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946648 2571 flags.go:64] FLAG: --reserved-cpus="" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946651 2571 flags.go:64] FLAG: --reserved-memory="" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946655 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946658 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946661 2571 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946664 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946667 2571 flags.go:64] FLAG: --runonce="false" Jul 09 09:54:04.954256 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946670 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946673 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946676 2571 flags.go:64] FLAG: --seccomp-default="false" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946679 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946681 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946684 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946687 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946690 2571 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946693 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946696 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946699 2571 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946702 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946705 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946708 2571 flags.go:64] FLAG: --system-cgroups="" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946711 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946716 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946720 2571 flags.go:64] FLAG: --tls-cert-file="" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946723 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946727 2571 flags.go:64] FLAG: --tls-min-version="" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946730 2571 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946732 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946735 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946739 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946742 2571 flags.go:64] FLAG: --v="2" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946746 2571 flags.go:64] FLAG: --version="false" Jul 09 09:54:04.954873 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946750 2571 flags.go:64] FLAG: --vmodule="" Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946753 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.946757 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946886 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946891 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946894 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946897 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946900 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946903 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946906 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946909 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946912 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946915 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946917 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946920 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946923 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946925 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946928 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946930 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946933 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:54:04.955500 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946935 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946938 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946941 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946947 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946949 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946952 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946954 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946957 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946959 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946963 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946966 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946969 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946984 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946987 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946991 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946994 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946996 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.946999 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947002 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947005 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:54:04.955987 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947007 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947010 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947013 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947016 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947018 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947020 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947023 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947025 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947028 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947030 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947033 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947035 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947038 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947040 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947043 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947047 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947050 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947052 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947055 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947057 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:54:04.956606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947060 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947064 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947066 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947069 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947071 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947074 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947076 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947079 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947081 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947086 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947089 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947092 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947095 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947098 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947100 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947103 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947106 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947110 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:54:04.957098 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947114 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947118 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947134 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947137 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947140 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947143 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947146 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947149 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947151 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947157 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947160 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947162 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947164 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947167 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947170 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947173 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947176 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947178 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947181 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:54:04.957553 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947184 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947187 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947189 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947192 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.947194 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.947199 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.953797 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.953811 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.954602 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.954610 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954663 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954667 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954671 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954674 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954677 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:54:04.958043 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954679 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954682 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954684 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954689 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954692 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954696 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954699 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954702 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954705 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954708 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954711 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954713 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954716 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954719 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954721 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954724 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954726 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954729 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954732 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:54:04.958448 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954734 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954737 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954739 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954741 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954744 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954749 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954752 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954754 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954757 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954760 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954762 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954765 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954767 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954770 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954772 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954775 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954777 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954781 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954785 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954788 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:54:04.958897 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954791 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954794 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954796 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954799 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954801 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954804 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954806 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954809 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954811 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954814 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954816 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954818 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954821 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954823 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954826 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954829 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954831 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954833 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954836 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954839 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:54:04.959385 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954841 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954844 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954846 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954850 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954854 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954857 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954859 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954862 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954864 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954867 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954869 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954871 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954874 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954877 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954880 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954882 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954885 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954887 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954890 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:54:04.959918 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954893 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954896 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954899 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954901 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954903 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954906 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954908 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954911 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954913 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954916 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954918 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954921 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954924 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954926 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954929 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:54:04.960743 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.954931 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.954936 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955044 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955050 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955054 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955058 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955061 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955064 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955067 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955070 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955073 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955075 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955078 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955081 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 09:54:04.961340 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955083 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955085 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955088 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955091 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955093 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955095 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955098 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955100 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955103 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955105 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955108 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955110 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955113 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955115 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955118 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955133 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955138 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955141 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955143 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955145 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955148 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 09:54:04.961784 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955151 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955153 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955156 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955158 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955161 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955163 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955166 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955168 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955171 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955174 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955176 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955179 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955181 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955184 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955187 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955189 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955192 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955194 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955196 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955199 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 09:54:04.962310 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955201 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955204 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955207 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955210 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955213 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955216 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955219 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955222 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955225 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955227 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955230 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955232 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955234 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955237 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955239 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955241 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955244 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955247 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955249 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 09:54:04.962831 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955252 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955254 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955257 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955259 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955262 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955264 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955267 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955269 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955272 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955274 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955277 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955279 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955282 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955284 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955288 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955291 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955294 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955296 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955299 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955301 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 09:54:04.963392 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955303 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955306 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955308 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955311 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955313 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955316 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:04.955318 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.955323 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.955422 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.957837 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.959316 2571 server.go:1019] "Starting client certificate rotation" Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.959412 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.959448 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 09:54:04.964012 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.963191 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 09:54:04.964758 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.964742 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 09:54:04.974590 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.974573 2571 log.go:25] "Validated CRI v1 runtime API" Jul 09 09:54:04.978180 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.978164 2571 log.go:25] "Validated CRI v1 image API" Jul 09 09:54:04.979511 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.979496 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 09:54:04.983592 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.983575 2571 fs.go:135] Filesystem UUIDs: map[16949886-efbd-469c-b68a-3d7a87979789:/dev/nvme0n1p4 39d366b3-76e6-4cf6-9892-05b16f569821:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 09 09:54:04.983643 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.983592 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 09:54:04.988577 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.988467 2571 manager.go:216] Machine: {Timestamp:2026-07-09 09:54:04.987162884 +0000 UTC m=+0.319646673 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097454 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229105a8a6e08224f4deb4f6105135 SystemUUID:ec229105-a8a6-e082-24f4-deb4f6105135 BootID:78479548-37ec-4170-96dc-8979e87324cd Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:52:ac:c8:8e:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:52:ac:c8:8e:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:a1:c6:5a:81:ca Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 09:54:04.988577 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.988572 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 09:54:04.988700 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.988646 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 09:54:04.989707 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.989682 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 09:54:04.989837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.989711 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-69.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 09:54:04.991055 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.991044 2571 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 09:54:04.991093 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.991058 2571 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 09:54:04.991093 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.991065 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 09:54:04.991775 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.991764 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 09:54:04.992360 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.992342 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 09:54:04.993285 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.993273 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 09:54:04.997419 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997406 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 09 09:54:04.997535 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997526 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 09:54:04.997576 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997557 2571 kubelet.go:486] "Attempting to sync node with API server" Jul 09 09:54:04.997576 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997571 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 09:54:04.997627 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997585 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 09:54:04.997627 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997593 2571 kubelet.go:398] "Adding apiserver pod source" Jul 09 09:54:04.997627 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997602 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 09:54:04.997979 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997957 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 09:54:04.998041 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:04.997981 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 09:54:05.001531 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.001513 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 09:54:05.003513 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.003500 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 09:54:05.003565 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.003530 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 09:54:05.004646 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004635 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 09:54:05.004680 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004651 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 09:54:05.004680 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004658 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 09:54:05.004680 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004664 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 09:54:05.004680 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004672 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 09:54:05.004680 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004678 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 09:54:05.004813 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004686 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 09:54:05.004813 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004692 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 09:54:05.004813 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004700 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 09:54:05.004813 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004705 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 09:54:05.004813 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004714 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 09:54:05.004813 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.004719 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 09:54:05.005531 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.005521 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 09:54:05.005531 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.005531 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 09:54:05.007416 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.007392 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 09:54:05.007469 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.007399 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-69.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 09:54:05.007503 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.007479 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-69.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 09:54:05.008839 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.008826 2571 server.go:1268] "Started kubelet" Jul 09 09:54:05.008933 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.008880 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 09:54:05.009033 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.008950 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 09:54:05.009356 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.009336 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 09:54:05.009634 ip-10-0-140-69 systemd[1]: Started Kubernetes Kubelet. Jul 09 09:54:05.010560 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.010544 2571 server.go:310] "Adding debug handlers to kubelet server" Jul 09 09:54:05.011061 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.011043 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 09:54:05.016261 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.016244 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 09:54:05.016659 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.013651 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-69.ec2.internal.18c0966c756319ec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-69.ec2.internal,UID:ip-10-0-140-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-69.ec2.internal,},FirstTimestamp:2026-07-09 09:54:05.008779756 +0000 UTC m=+0.341263544,LastTimestamp:2026-07-09 09:54:05.008779756 +0000 UTC m=+0.341263544,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-69.ec2.internal,}" Jul 09 09:54:05.017016 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.016999 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 09:54:05.017016 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.017012 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 09:54:05.017136 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.017073 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.017607 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.017590 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 09:54:05.017681 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.017643 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 09:54:05.017681 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.017660 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 09:54:05.017794 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.017774 2571 reconstruct.go:97] "Volume reconstruction finished" Jul 09 09:54:05.017794 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.017789 2571 reconciler.go:29] "Reconciler: start to sync state" Jul 09 09:54:05.017943 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.017820 2571 factory.go:55] Registering systemd factory Jul 09 09:54:05.017943 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.017856 2571 factory.go:223] Registration of the systemd container factory successfully Jul 09 09:54:05.018055 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.018031 2571 factory.go:153] Registering CRI-O factory Jul 09 09:54:05.018055 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.018041 2571 factory.go:223] Registration of the crio container factory successfully Jul 09 09:54:05.018153 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.018082 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 09:54:05.018153 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.018104 2571 factory.go:103] Registering Raw factory Jul 09 09:54:05.018245 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.018117 2571 manager.go:1196] Started watching for new ooms in manager Jul 09 09:54:05.019033 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.019019 2571 manager.go:349] Starting recovery of all containers Jul 09 09:54:05.024300 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.023945 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 09:54:05.027868 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.027838 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 09:54:05.028752 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.028725 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-69.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 09:54:05.032911 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.032897 2571 manager.go:354] Recovery completed Jul 09 09:54:05.038946 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.038934 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:54:05.041383 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.041360 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:54:05.041460 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.041398 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:54:05.041460 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.041412 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:54:05.041995 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.041979 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:54:05.042070 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.041979 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 09:54:05.042070 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.042042 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 09:54:05.042070 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.042059 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 09 09:54:05.042854 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.042836 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:54:05.042933 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.042866 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:54:05.042933 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.042882 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:54:05.043220 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.043141 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-69.ec2.internal.18c0966c7754914f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-69.ec2.internal,UID:ip-10-0-140-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-69.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-69.ec2.internal,},FirstTimestamp:2026-07-09 09:54:05.041381711 +0000 UTC m=+0.373865501,LastTimestamp:2026-07-09 09:54:05.041381711 +0000 UTC m=+0.373865501,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-69.ec2.internal,}" Jul 09 09:54:05.044286 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.044269 2571 policy_none.go:49] "None policy: Start" Jul 09 09:54:05.044286 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.044286 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 09:54:05.044412 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.044296 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 09:54:05.046265 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.046252 2571 policy_none.go:47] "Start" Jul 09 09:54:05.054673 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.054436 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-69.ec2.internal.18c0966c7754eb6a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-69.ec2.internal,UID:ip-10-0-140-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-69.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-69.ec2.internal,},FirstTimestamp:2026-07-09 09:54:05.041404778 +0000 UTC m=+0.373888569,LastTimestamp:2026-07-09 09:54:05.041404778 +0000 UTC m=+0.373888569,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-69.ec2.internal,}" Jul 09 09:54:05.055381 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.055362 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k5jhx" Jul 09 09:54:05.062903 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.062879 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-k5jhx" Jul 09 09:54:05.064777 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.064687 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-69.ec2.internal.18c0966c775520a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-69.ec2.internal,UID:ip-10-0-140-69.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-69.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-69.ec2.internal,},FirstTimestamp:2026-07-09 09:54:05.041418404 +0000 UTC m=+0.373902194,LastTimestamp:2026-07-09 09:54:05.041418404 +0000 UTC m=+0.373902194,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-69.ec2.internal,}" Jul 09 09:54:05.066770 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.066757 2571 manager.go:337] "Starting Device Plugin manager" Jul 09 09:54:05.066831 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.066799 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 09:54:05.066831 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.066811 2571 server.go:87] "Starting device plugin registration server" Jul 09 09:54:05.067053 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.067041 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 09:54:05.067166 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.067054 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 09:54:05.067218 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.067159 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 09:54:05.067274 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.067241 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 09:54:05.067274 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.067250 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 09:54:05.067732 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.067715 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 09:54:05.067809 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.067764 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.153665 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.153642 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 09:54:05.153836 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.153682 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 09:54:05.153836 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.153702 2571 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 09:54:05.153836 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.153733 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 09:54:05.156315 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.156287 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:54:05.168060 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.168038 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:54:05.168961 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.168946 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:54:05.169050 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.168975 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:54:05.169050 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.168994 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:54:05.169050 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.169022 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.184395 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.184378 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.184463 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.184401 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-69.ec2.internal\": node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.210324 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.210269 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.254021 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.253983 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal"] Jul 09 09:54:05.254157 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.254075 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:54:05.254963 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.254946 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:54:05.255032 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.254975 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:54:05.255032 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.254992 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:54:05.257364 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.257351 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:54:05.257533 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.257517 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.257599 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.257578 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:54:05.258048 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.258024 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:54:05.258149 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.258033 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:54:05.258149 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.258084 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:54:05.258149 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.258095 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:54:05.258149 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.258059 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:54:05.258149 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.258138 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:54:05.260255 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.260239 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.260323 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.260264 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 09:54:05.260829 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.260814 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientMemory" Jul 09 09:54:05.260909 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.260838 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 09:54:05.260909 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.260852 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeHasSufficientPID" Jul 09 09:54:05.274697 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.274679 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-69.ec2.internal\" not found" node="ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.278473 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.278457 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-69.ec2.internal\" not found" node="ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.310505 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.310482 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.318552 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.318528 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aacd596fa6102c9a13bb44707f0c6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal\" (UID: \"6aacd596fa6102c9a13bb44707f0c6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.318620 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.318557 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5d7380ed3c9d0d9e6311c53297db640-config\") pod \"kube-apiserver-proxy-ip-10-0-140-69.ec2.internal\" (UID: \"f5d7380ed3c9d0d9e6311c53297db640\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.318620 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.318573 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6aacd596fa6102c9a13bb44707f0c6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal\" (UID: \"6aacd596fa6102c9a13bb44707f0c6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.410701 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.410677 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.418942 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.418925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6aacd596fa6102c9a13bb44707f0c6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal\" (UID: \"6aacd596fa6102c9a13bb44707f0c6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.418987 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.418949 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aacd596fa6102c9a13bb44707f0c6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal\" (UID: \"6aacd596fa6102c9a13bb44707f0c6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.418987 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.418970 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5d7380ed3c9d0d9e6311c53297db640-config\") pod \"kube-apiserver-proxy-ip-10-0-140-69.ec2.internal\" (UID: \"f5d7380ed3c9d0d9e6311c53297db640\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.419060 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.418996 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f5d7380ed3c9d0d9e6311c53297db640-config\") pod \"kube-apiserver-proxy-ip-10-0-140-69.ec2.internal\" (UID: \"f5d7380ed3c9d0d9e6311c53297db640\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.419060 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.419026 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aacd596fa6102c9a13bb44707f0c6b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal\" (UID: \"6aacd596fa6102c9a13bb44707f0c6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.419060 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.419024 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6aacd596fa6102c9a13bb44707f0c6b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal\" (UID: \"6aacd596fa6102c9a13bb44707f0c6b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.511353 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.511277 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.577749 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.577713 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.581494 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.581474 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" Jul 09 09:54:05.612075 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.612036 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.712601 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.712564 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.814748 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.814658 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.916524 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:05.916483 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:05.960475 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.960441 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 09:54:05.961007 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:05.960599 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 09:54:06.017462 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:06.017423 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-69.ec2.internal\" not found" Jul 09 09:54:06.052628 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.052594 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:54:06.065901 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.065804 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 09:49:05 +0000 UTC" deadline="2028-02-28 07:19:21.03293381 +0000 UTC" Jul 09 09:54:06.065901 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.065837 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14373h25m14.967100988s" Jul 09 09:54:06.103682 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.103651 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:54:06.117215 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.117191 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" Jul 09 09:54:06.134230 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.134199 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 09:54:06.135247 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.135232 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" Jul 09 09:54:06.147071 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.147049 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 09:54:06.360514 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:06.360480 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aacd596fa6102c9a13bb44707f0c6b4.slice/crio-914cc1c6ffa37501b8abbf5c232e05daaf3e4cac6dd2c4d9cce5265ae72b06d0 WatchSource:0}: Error finding container 914cc1c6ffa37501b8abbf5c232e05daaf3e4cac6dd2c4d9cce5265ae72b06d0: Status 404 returned error can't find the container with id 914cc1c6ffa37501b8abbf5c232e05daaf3e4cac6dd2c4d9cce5265ae72b06d0 Jul 09 09:54:06.360817 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:06.360792 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d7380ed3c9d0d9e6311c53297db640.slice/crio-b9ae703c8e474fc327226354622404563bbccc6b5d6ba7ee26dede9b7ee5839c WatchSource:0}: Error finding container b9ae703c8e474fc327226354622404563bbccc6b5d6ba7ee26dede9b7ee5839c: Status 404 returned error can't find the container with id b9ae703c8e474fc327226354622404563bbccc6b5d6ba7ee26dede9b7ee5839c Jul 09 09:54:06.365089 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.365075 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 09:54:06.534483 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.534455 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:54:06.905294 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.905261 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 09:54:06.998709 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:06.998668 2571 apiserver.go:52] "Watching apiserver" Jul 09 09:54:07.010278 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.010247 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 09:54:07.011503 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.011468 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh","openshift-image-registry/node-ca-l84jq","openshift-multus/network-metrics-daemon-8kpn9","openshift-network-diagnostics/network-check-target-w8zqv","openshift-ovn-kubernetes/ovnkube-node-b47sp","kube-system/global-pull-secret-syncer-slmhl","kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dm2pn","openshift-dns/node-resolver-k57ns","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal","openshift-multus/multus-additional-cni-plugins-2mcbb","openshift-multus/multus-w7xn2","openshift-network-operator/iptables-alerter-9rbg7","kube-system/konnectivity-agent-tkpgz"] Jul 09 09:54:07.014284 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.014257 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.016650 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.016624 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.017577 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.017401 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 09:54:07.017577 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.017453 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 09:54:07.017577 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.017515 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:54:07.017915 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.017640 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 09:54:07.018211 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.018012 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h886w\"" Jul 09 09:54:07.019172 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.019126 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:07.019275 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.019256 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:07.019560 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.019538 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 09:54:07.019654 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.019567 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5l75b\"" Jul 09 09:54:07.019717 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.019555 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 09:54:07.019784 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.019766 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 09:54:07.021845 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.021809 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:07.021936 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.021884 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:07.024459 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.024439 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.027641 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.027620 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 09:54:07.027765 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.027736 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.027862 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.027838 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:07.027928 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.027880 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 09:54:07.028113 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028075 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwp4\" (UniqueName: \"kubernetes.io/projected/49062adc-752f-4d87-aa95-db7f44411a17-kube-api-access-hzwp4\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:07.028202 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028152 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d31879ac-51f8-4f04-8591-dd2c0487c68b-host\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.028202 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028161 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 09:54:07.028202 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028184 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d31879ac-51f8-4f04-8591-dd2c0487c68b-serviceca\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.028369 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028240 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-iptables-alerter-script\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.028369 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028275 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-host-slash\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.028475 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028394 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcdwj\" (UniqueName: \"kubernetes.io/projected/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-kube-api-access-fcdwj\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.028475 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkpdk\" (UniqueName: \"kubernetes.io/projected/d31879ac-51f8-4f04-8591-dd2c0487c68b-kube-api-access-jkpdk\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.028475 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028459 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:07.028634 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.028592 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 09:54:07.029628 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.029607 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8rnsd\"" Jul 09 09:54:07.031323 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.030041 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 09:54:07.031323 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.030249 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 09:54:07.031601 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.031577 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 09:54:07.032821 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.032799 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.035597 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.035364 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 09:54:07.035597 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.035416 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.035597 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.035507 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 09:54:07.035805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.035738 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 09:54:07.035869 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.035854 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n4brl\"" Jul 09 09:54:07.038043 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.038023 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:54:07.038288 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.038270 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.038406 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.038381 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-csfhq\"" Jul 09 09:54:07.038406 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.038395 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 09:54:07.040654 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.040634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:07.041115 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.041095 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 09:54:07.041238 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.041196 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z2js2\"" Jul 09 09:54:07.041355 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.041336 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 09:54:07.043328 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.043300 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5wlx7\"" Jul 09 09:54:07.043777 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.043756 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.043870 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.043799 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 09:54:07.044553 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.044402 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 09:54:07.046476 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.046455 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.047522 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.047502 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 09:54:07.047630 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.047598 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 09:54:07.047745 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.047726 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 09:54:07.047996 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.047979 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w4h5c\"" Jul 09 09:54:07.048057 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.048041 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 09:54:07.048187 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.047980 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 09:54:07.049164 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.049145 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v9q5z\"" Jul 09 09:54:07.049274 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.049244 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 09:54:07.058915 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.058891 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x52tw" Jul 09 09:54:07.068687 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.068666 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x52tw" Jul 09 09:54:07.120279 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.119157 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 09:54:07.128935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.128720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-cni-bin\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.128935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.128764 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-ovn\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.128935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.128791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-ovnkube-script-lib\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.128935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.128818 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysconfig\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.128935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.128845 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-tuned\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.128935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.128868 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-ovnkube-config\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.128935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.128892 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysctl-d\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.129419 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.128944 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcdwj\" (UniqueName: \"kubernetes.io/projected/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-kube-api-access-fcdwj\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.129419 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129314 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-cni-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.129419 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-cnibin\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.129419 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129414 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-os-release\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.129615 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129437 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-hostroot\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.129615 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129462 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-cni-netd\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.129615 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-etc-kubernetes\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.129615 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5181675-9c76-4ae5-bb57-dd4c8558b80f-agent-certs\") pod \"konnectivity-agent-tkpgz\" (UID: \"b5181675-9c76-4ae5-bb57-dd4c8558b80f\") " pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:07.129615 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129538 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5181675-9c76-4ae5-bb57-dd4c8558b80f-konnectivity-ca\") pod \"konnectivity-agent-tkpgz\" (UID: \"b5181675-9c76-4ae5-bb57-dd4c8558b80f\") " pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:07.129615 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129583 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-os-release\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129638 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-sys-fs\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129666 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b75e24a-dd42-4539-9577-707efd61daaa-cni-binary-copy\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-kubelet\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129721 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-var-lib-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129745 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/048fd61f-ba5b-41bc-95a5-366a5734571d-dbus\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129792 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-socket-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129817 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129851 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-modprobe-d\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.129899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129893 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-hosts-file\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129917 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-conf-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129942 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-cni-bin\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.129996 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-device-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-var-lib-kubelet\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130045 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-host\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkpdk\" (UniqueName: \"kubernetes.io/projected/d31879ac-51f8-4f04-8591-dd2c0487c68b-kube-api-access-jkpdk\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130107 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-system-cni-dir\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130197 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-run\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130233 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-tmp\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130262 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-system-cni-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130278 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-slash\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130301 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-run-netns\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130326 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-systemd\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.130423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-cnibin\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/048fd61f-ba5b-41bc-95a5-366a5734571d-kubelet-config\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130396 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130428 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d31879ac-51f8-4f04-8591-dd2c0487c68b-serviceca\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130453 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-cni-multus\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130478 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-run-ovn-kubernetes\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-env-overrides\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130562 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d41af354-3b0c-4774-98df-f67897540b38-ovn-node-metrics-cert\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130590 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130616 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-systemd\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130639 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-lib-modules\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130666 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130716 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-sys\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130739 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-tmp-dir\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130775 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpgwh\" (UniqueName: \"kubernetes.io/projected/1b75e24a-dd42-4539-9577-707efd61daaa-kube-api-access-zpgwh\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.131243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130799 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-systemd-units\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130844 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-node-log\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130880 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmkz\" (UniqueName: \"kubernetes.io/projected/1090cf01-53a3-486e-9641-90a2ca359480-kube-api-access-9nmkz\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130908 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysctl-conf\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wth2g\" (UniqueName: \"kubernetes.io/projected/d41af354-3b0c-4774-98df-f67897540b38-kube-api-access-wth2g\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-iptables-alerter-script\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.130995 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b75e24a-dd42-4539-9577-707efd61daaa-multus-daemon-config\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131045 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131070 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-etc-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131094 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-registration-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131118 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-kubernetes\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d31879ac-51f8-4f04-8591-dd2c0487c68b-host\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131186 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txrvs\" (UniqueName: \"kubernetes.io/projected/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-kube-api-access-txrvs\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131209 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-k8s-cni-cncf-io\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131234 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bff\" (UniqueName: \"kubernetes.io/projected/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-kube-api-access-w5bff\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.132134 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131261 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-host-slash\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwp4\" (UniqueName: \"kubernetes.io/projected/49062adc-752f-4d87-aa95-db7f44411a17-kube-api-access-hzwp4\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131317 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-multus-certs\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131345 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-log-socket\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8xn\" (UniqueName: \"kubernetes.io/projected/97460b30-436a-49e4-9a03-b6ed61a25ca5-kube-api-access-hs8xn\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131397 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-socket-dir-parent\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131422 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-netns\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131446 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-kubelet\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131470 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131597 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-iptables-alerter-script\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.131628 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131649 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-host-slash\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.131691 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d31879ac-51f8-4f04-8591-dd2c0487c68b-host\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.131723 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs podName:49062adc-752f-4d87-aa95-db7f44411a17 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:07.631685292 +0000 UTC m=+2.964169074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs") pod "network-metrics-daemon-8kpn9" (UID: "49062adc-752f-4d87-aa95-db7f44411a17") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:07.132930 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.132115 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d31879ac-51f8-4f04-8591-dd2c0487c68b-serviceca\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.138127 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.138101 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 09:54:07.143064 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.143036 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkpdk\" (UniqueName: \"kubernetes.io/projected/d31879ac-51f8-4f04-8591-dd2c0487c68b-kube-api-access-jkpdk\") pod \"node-ca-l84jq\" (UID: \"d31879ac-51f8-4f04-8591-dd2c0487c68b\") " pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.143600 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.143558 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwp4\" (UniqueName: \"kubernetes.io/projected/49062adc-752f-4d87-aa95-db7f44411a17-kube-api-access-hzwp4\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:07.144681 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.144650 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcdwj\" (UniqueName: \"kubernetes.io/projected/08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6-kube-api-access-fcdwj\") pod \"iptables-alerter-9rbg7\" (UID: \"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6\") " pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.160351 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.160262 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" event={"ID":"f5d7380ed3c9d0d9e6311c53297db640","Type":"ContainerStarted","Data":"b9ae703c8e474fc327226354622404563bbccc6b5d6ba7ee26dede9b7ee5839c"} Jul 09 09:54:07.161943 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.161913 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" event={"ID":"6aacd596fa6102c9a13bb44707f0c6b4","Type":"ContainerStarted","Data":"914cc1c6ffa37501b8abbf5c232e05daaf3e4cac6dd2c4d9cce5265ae72b06d0"} Jul 09 09:54:07.232679 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-cni-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.232679 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-cnibin\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232709 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-os-release\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-hostroot\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232772 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-cni-netd\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232794 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-etc-kubernetes\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5181675-9c76-4ae5-bb57-dd4c8558b80f-agent-certs\") pod \"konnectivity-agent-tkpgz\" (UID: \"b5181675-9c76-4ae5-bb57-dd4c8558b80f\") " pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232853 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5181675-9c76-4ae5-bb57-dd4c8558b80f-konnectivity-ca\") pod \"konnectivity-agent-tkpgz\" (UID: \"b5181675-9c76-4ae5-bb57-dd4c8558b80f\") " pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232879 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-os-release\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.232903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-sys-fs\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233030 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-cnibin\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233083 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b75e24a-dd42-4539-9577-707efd61daaa-cni-binary-copy\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-kubelet\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233116 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-os-release\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.233118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233139 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-var-lib-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233178 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-etc-kubernetes\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233205 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-os-release\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233231 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/048fd61f-ba5b-41bc-95a5-366a5734571d-dbus\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233258 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-sys-fs\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233263 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-socket-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233316 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-modprobe-d\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233343 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-hosts-file\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233368 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-conf-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233392 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-cni-bin\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-device-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-var-lib-kubelet\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-host\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233534 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5181675-9c76-4ae5-bb57-dd4c8558b80f-konnectivity-ca\") pod \"konnectivity-agent-tkpgz\" (UID: \"b5181675-9c76-4ae5-bb57-dd4c8558b80f\") " pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:07.233805 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233557 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-system-cni-dir\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233566 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-hosts-file\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233611 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-run\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233633 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-tmp\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233666 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-system-cni-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233672 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-socket-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233679 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-etc-selinux\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233691 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-slash\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233715 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-run-netns\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233745 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-host\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233750 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/048fd61f-ba5b-41bc-95a5-366a5734571d-dbus\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233753 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-systemd\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233774 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-modprobe-d\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233801 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-cni-bin\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233810 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-system-cni-dir\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233816 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b75e24a-dd42-4539-9577-707efd61daaa-cni-binary-copy\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233835 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-cnibin\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.234675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233851 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-systemd\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233862 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233880 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-kubelet\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233864 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/048fd61f-ba5b-41bc-95a5-366a5734571d-kubelet-config\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233895 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-conf-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233914 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/048fd61f-ba5b-41bc-95a5-366a5734571d-kubelet-config\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233913 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-cnibin\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233959 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-system-cni-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233970 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-run\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233970 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-var-lib-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.233990 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-cni-multus\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234014 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-run-netns\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234023 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-cni-multus\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234029 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-run-ovn-kubernetes\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234059 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-env-overrides\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234061 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1090cf01-53a3-486e-9641-90a2ca359480-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234069 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-run-ovn-kubernetes\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.235668 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.234030 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234070 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-var-lib-kubelet\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234085 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d41af354-3b0c-4774-98df-f67897540b38-ovn-node-metrics-cert\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234088 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-cni-netd\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234060 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-slash\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234116 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-device-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.234145 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret podName:048fd61f-ba5b-41bc-95a5-366a5734571d nodeName:}" failed. No retries permitted until 2026-07-09 09:54:07.734125625 +0000 UTC m=+3.066609420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret") pod "global-pull-secret-syncer-slmhl" (UID: "048fd61f-ba5b-41bc-95a5-366a5734571d") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234250 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-hostroot\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234326 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-systemd\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234378 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-systemd\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234378 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234393 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-cni-dir\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234428 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-lib-modules\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234485 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234513 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-sys\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.236575 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234536 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-tmp-dir\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234564 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpgwh\" (UniqueName: \"kubernetes.io/projected/1b75e24a-dd42-4539-9577-707efd61daaa-kube-api-access-zpgwh\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234566 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-lib-modules\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234603 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-systemd-units\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234618 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-cni-binary-copy\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234630 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-node-log\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmkz\" (UniqueName: \"kubernetes.io/projected/1090cf01-53a3-486e-9641-90a2ca359480-kube-api-access-9nmkz\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234686 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysctl-conf\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wth2g\" (UniqueName: \"kubernetes.io/projected/d41af354-3b0c-4774-98df-f67897540b38-kube-api-access-wth2g\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b75e24a-dd42-4539-9577-707efd61daaa-multus-daemon-config\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234814 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-etc-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234842 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-registration-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234808 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-env-overrides\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-kubernetes\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234899 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txrvs\" (UniqueName: \"kubernetes.io/projected/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-kube-api-access-txrvs\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234904 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.237442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-k8s-cni-cncf-io\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234951 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bff\" (UniqueName: \"kubernetes.io/projected/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-kube-api-access-w5bff\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234980 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-node-log\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.234983 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-multus-certs\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235022 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-multus-certs\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235029 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-log-socket\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235059 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8xn\" (UniqueName: \"kubernetes.io/projected/97460b30-436a-49e4-9a03-b6ed61a25ca5-kube-api-access-hs8xn\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235061 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1090cf01-53a3-486e-9641-90a2ca359480-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-socket-dir-parent\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235130 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-sys\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235132 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-netns\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235166 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-netns\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235172 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-kubelet\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235226 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-systemd-units\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235228 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235263 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235280 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysctl-conf\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-cni-bin\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.238293 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235326 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-var-lib-cni-bin\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235330 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-ovn\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235363 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-ovnkube-script-lib\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysconfig\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-tuned\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235445 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-ovnkube-config\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235471 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysctl-d\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235540 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1b75e24a-dd42-4539-9577-707efd61daaa-multus-daemon-config\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235562 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-host-kubelet\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235603 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysctl-d\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235647 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-run-ovn\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235680 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-tmp-dir\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235685 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-sysconfig\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235365 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-log-socket\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235717 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d41af354-3b0c-4774-98df-f67897540b38-etc-openvswitch\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235742 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-multus-socket-dir-parent\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235772 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-kubernetes\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235782 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97460b30-436a-49e4-9a03-b6ed61a25ca5-registration-dir\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.239231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.235829 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1b75e24a-dd42-4539-9577-707efd61daaa-host-run-k8s-cni-cncf-io\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.240232 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.236327 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-ovnkube-script-lib\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.240232 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.236381 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d41af354-3b0c-4774-98df-f67897540b38-ovnkube-config\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.240232 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.238641 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-etc-tuned\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.240232 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.239027 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5181675-9c76-4ae5-bb57-dd4c8558b80f-agent-certs\") pod \"konnectivity-agent-tkpgz\" (UID: \"b5181675-9c76-4ae5-bb57-dd4c8558b80f\") " pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:07.240617 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.240594 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-tmp\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.241277 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.241253 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d41af354-3b0c-4774-98df-f67897540b38-ovn-node-metrics-cert\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.244948 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.244923 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:54:07.245102 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.244954 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:54:07.245102 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.244968 2571 projected.go:196] Error preparing data for projected volume kube-api-access-nhtrb for pod openshift-network-diagnostics/network-check-target-w8zqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:07.245102 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.245064 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb podName:2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf nodeName:}" failed. No retries permitted until 2026-07-09 09:54:07.745045217 +0000 UTC m=+3.077528995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nhtrb" (UniqueName: "kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb") pod "network-check-target-w8zqv" (UID: "2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:07.246544 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.246517 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmkz\" (UniqueName: \"kubernetes.io/projected/1090cf01-53a3-486e-9641-90a2ca359480-kube-api-access-9nmkz\") pod \"multus-additional-cni-plugins-2mcbb\" (UID: \"1090cf01-53a3-486e-9641-90a2ca359480\") " pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.247502 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.247458 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8xn\" (UniqueName: \"kubernetes.io/projected/97460b30-436a-49e4-9a03-b6ed61a25ca5-kube-api-access-hs8xn\") pod \"aws-ebs-csi-driver-node-t4mkh\" (UID: \"97460b30-436a-49e4-9a03-b6ed61a25ca5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.247904 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.247880 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wth2g\" (UniqueName: \"kubernetes.io/projected/d41af354-3b0c-4774-98df-f67897540b38-kube-api-access-wth2g\") pod \"ovnkube-node-b47sp\" (UID: \"d41af354-3b0c-4774-98df-f67897540b38\") " pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.248932 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.248875 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txrvs\" (UniqueName: \"kubernetes.io/projected/8ff1e23f-f24b-4568-b9ee-1f0692a9afe3-kube-api-access-txrvs\") pod \"node-resolver-k57ns\" (UID: \"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3\") " pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.249243 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.249224 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bff\" (UniqueName: \"kubernetes.io/projected/13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e-kube-api-access-w5bff\") pod \"tuned-dm2pn\" (UID: \"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e\") " pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.250371 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.250349 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpgwh\" (UniqueName: \"kubernetes.io/projected/1b75e24a-dd42-4539-9577-707efd61daaa-kube-api-access-zpgwh\") pod \"multus-w7xn2\" (UID: \"1b75e24a-dd42-4539-9577-707efd61daaa\") " pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.329926 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.329888 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9rbg7" Jul 09 09:54:07.337675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.337651 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l84jq" Jul 09 09:54:07.342606 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.342575 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c3e090_2c2d_4552_8d3b_5fb7c7cfc4b6.slice/crio-b9dd3ed2b1724041453d8d4d00ca33c86375607b112195afcb79faf110906cd2 WatchSource:0}: Error finding container b9dd3ed2b1724041453d8d4d00ca33c86375607b112195afcb79faf110906cd2: Status 404 returned error can't find the container with id b9dd3ed2b1724041453d8d4d00ca33c86375607b112195afcb79faf110906cd2 Jul 09 09:54:07.347728 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.347701 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31879ac_51f8_4f04_8591_dd2c0487c68b.slice/crio-4b7e06b4f360bf6492e00d91a89212049764b230f29cc6dbaf857570db7bc48c WatchSource:0}: Error finding container 4b7e06b4f360bf6492e00d91a89212049764b230f29cc6dbaf857570db7bc48c: Status 404 returned error can't find the container with id 4b7e06b4f360bf6492e00d91a89212049764b230f29cc6dbaf857570db7bc48c Jul 09 09:54:07.349139 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.349102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:07.356281 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.356258 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" Jul 09 09:54:07.359820 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.359796 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41af354_3b0c_4774_98df_f67897540b38.slice/crio-805e27965aac6fae9f66dd7213097a9844d30097d75b4da3766009f1173f96ba WatchSource:0}: Error finding container 805e27965aac6fae9f66dd7213097a9844d30097d75b4da3766009f1173f96ba: Status 404 returned error can't find the container with id 805e27965aac6fae9f66dd7213097a9844d30097d75b4da3766009f1173f96ba Jul 09 09:54:07.362878 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.362786 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" Jul 09 09:54:07.366436 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.366414 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97460b30_436a_49e4_9a03_b6ed61a25ca5.slice/crio-24e4589544da529464846dab63879d3a6fd4b1c4559bb461d105c5b402df24b8 WatchSource:0}: Error finding container 24e4589544da529464846dab63879d3a6fd4b1c4559bb461d105c5b402df24b8: Status 404 returned error can't find the container with id 24e4589544da529464846dab63879d3a6fd4b1c4559bb461d105c5b402df24b8 Jul 09 09:54:07.371501 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.371478 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k57ns" Jul 09 09:54:07.372309 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.372288 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a3ff5c_d58e_4dcf_af73_5129ab6bbe8e.slice/crio-b0d1528e2b35886d8389a9fd5f9b75e28298f685306f8b0572c6789072959d21 WatchSource:0}: Error finding container b0d1528e2b35886d8389a9fd5f9b75e28298f685306f8b0572c6789072959d21: Status 404 returned error can't find the container with id b0d1528e2b35886d8389a9fd5f9b75e28298f685306f8b0572c6789072959d21 Jul 09 09:54:07.379973 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.379948 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:07.383880 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.383855 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff1e23f_f24b_4568_b9ee_1f0692a9afe3.slice/crio-af943db28c54581a34fcfd7e7df76f6f1f29a52c5264583fd4212320f2889855 WatchSource:0}: Error finding container af943db28c54581a34fcfd7e7df76f6f1f29a52c5264583fd4212320f2889855: Status 404 returned error can't find the container with id af943db28c54581a34fcfd7e7df76f6f1f29a52c5264583fd4212320f2889855 Jul 09 09:54:07.386981 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.386960 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" Jul 09 09:54:07.393222 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.392625 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7xn2" Jul 09 09:54:07.393222 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.393186 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5181675_9c76_4ae5_bb57_dd4c8558b80f.slice/crio-15275f2942046b99fa4721fd96d48ebfacb970f7fed511a82e7d2e014d7fa1c7 WatchSource:0}: Error finding container 15275f2942046b99fa4721fd96d48ebfacb970f7fed511a82e7d2e014d7fa1c7: Status 404 returned error can't find the container with id 15275f2942046b99fa4721fd96d48ebfacb970f7fed511a82e7d2e014d7fa1c7 Jul 09 09:54:07.399507 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.399478 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1090cf01_53a3_486e_9641_90a2ca359480.slice/crio-9ac6cf85d4626e174ed3980ba6eb252b0e40f6579f332581dea46b4b94f450d3 WatchSource:0}: Error finding container 9ac6cf85d4626e174ed3980ba6eb252b0e40f6579f332581dea46b4b94f450d3: Status 404 returned error can't find the container with id 9ac6cf85d4626e174ed3980ba6eb252b0e40f6579f332581dea46b4b94f450d3 Jul 09 09:54:07.405443 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:07.405412 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b75e24a_dd42_4539_9577_707efd61daaa.slice/crio-0b84879f91735ac3d904c0fdc01c4129a03eb7a3421593c1c9477c10a6452c78 WatchSource:0}: Error finding container 0b84879f91735ac3d904c0fdc01c4129a03eb7a3421593c1c9477c10a6452c78: Status 404 returned error can't find the container with id 0b84879f91735ac3d904c0fdc01c4129a03eb7a3421593c1c9477c10a6452c78 Jul 09 09:54:07.639582 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.639468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:07.639771 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.639593 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:07.639771 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.639664 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs podName:49062adc-752f-4d87-aa95-db7f44411a17 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:08.639645122 +0000 UTC m=+3.972128899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs") pod "network-metrics-daemon-8kpn9" (UID: "49062adc-752f-4d87-aa95-db7f44411a17") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:07.740210 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.740176 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:07.740394 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.740362 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:07.740462 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.740452 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret podName:048fd61f-ba5b-41bc-95a5-366a5734571d nodeName:}" failed. No retries permitted until 2026-07-09 09:54:08.740425754 +0000 UTC m=+4.072909539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret") pod "global-pull-secret-syncer-slmhl" (UID: "048fd61f-ba5b-41bc-95a5-366a5734571d") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:07.840636 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:07.840577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:07.840830 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.840784 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:54:07.840830 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.840813 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:54:07.840830 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.840826 2571 projected.go:196] Error preparing data for projected volume kube-api-access-nhtrb for pod openshift-network-diagnostics/network-check-target-w8zqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:07.841049 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:07.840899 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb podName:2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf nodeName:}" failed. No retries permitted until 2026-07-09 09:54:08.840879081 +0000 UTC m=+4.173362868 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhtrb" (UniqueName: "kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb") pod "network-check-target-w8zqv" (UID: "2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:08.072325 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.072191 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 09:49:07 +0000 UTC" deadline="2028-04-01 07:49:55.231965004 +0000 UTC" Jul 09 09:54:08.072325 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.072258 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15165h55m47.159711308s" Jul 09 09:54:08.155181 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.155085 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:08.155381 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.155263 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:08.166244 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.166167 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7xn2" event={"ID":"1b75e24a-dd42-4539-9577-707efd61daaa","Type":"ContainerStarted","Data":"0b84879f91735ac3d904c0fdc01c4129a03eb7a3421593c1c9477c10a6452c78"} Jul 09 09:54:08.168508 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.168448 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tkpgz" event={"ID":"b5181675-9c76-4ae5-bb57-dd4c8558b80f","Type":"ContainerStarted","Data":"15275f2942046b99fa4721fd96d48ebfacb970f7fed511a82e7d2e014d7fa1c7"} Jul 09 09:54:08.170152 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.170114 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k57ns" event={"ID":"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3","Type":"ContainerStarted","Data":"af943db28c54581a34fcfd7e7df76f6f1f29a52c5264583fd4212320f2889855"} Jul 09 09:54:08.171808 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.171781 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" event={"ID":"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e","Type":"ContainerStarted","Data":"b0d1528e2b35886d8389a9fd5f9b75e28298f685306f8b0572c6789072959d21"} Jul 09 09:54:08.173154 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.173128 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" event={"ID":"97460b30-436a-49e4-9a03-b6ed61a25ca5","Type":"ContainerStarted","Data":"24e4589544da529464846dab63879d3a6fd4b1c4559bb461d105c5b402df24b8"} Jul 09 09:54:08.175074 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.175051 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"805e27965aac6fae9f66dd7213097a9844d30097d75b4da3766009f1173f96ba"} Jul 09 09:54:08.177010 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.176985 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" event={"ID":"1090cf01-53a3-486e-9641-90a2ca359480","Type":"ContainerStarted","Data":"9ac6cf85d4626e174ed3980ba6eb252b0e40f6579f332581dea46b4b94f450d3"} Jul 09 09:54:08.179109 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.179084 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l84jq" event={"ID":"d31879ac-51f8-4f04-8591-dd2c0487c68b","Type":"ContainerStarted","Data":"4b7e06b4f360bf6492e00d91a89212049764b230f29cc6dbaf857570db7bc48c"} Jul 09 09:54:08.180872 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.180848 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9rbg7" event={"ID":"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6","Type":"ContainerStarted","Data":"b9dd3ed2b1724041453d8d4d00ca33c86375607b112195afcb79faf110906cd2"} Jul 09 09:54:08.646126 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.646085 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:08.646356 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.646333 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:08.646431 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.646419 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs podName:49062adc-752f-4d87-aa95-db7f44411a17 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:10.646395404 +0000 UTC m=+5.978879195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs") pod "network-metrics-daemon-8kpn9" (UID: "49062adc-752f-4d87-aa95-db7f44411a17") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:08.746580 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.746535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:08.746776 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.746741 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:08.746857 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.746809 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret podName:048fd61f-ba5b-41bc-95a5-366a5734571d nodeName:}" failed. No retries permitted until 2026-07-09 09:54:10.746789793 +0000 UTC m=+6.079273586 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret") pod "global-pull-secret-syncer-slmhl" (UID: "048fd61f-ba5b-41bc-95a5-366a5734571d") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:08.847971 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:08.847235 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:08.847971 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.847502 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:54:08.847971 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.847526 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:54:08.847971 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.847541 2571 projected.go:196] Error preparing data for projected volume kube-api-access-nhtrb for pod openshift-network-diagnostics/network-check-target-w8zqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:08.847971 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:08.847601 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb podName:2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf nodeName:}" failed. No retries permitted until 2026-07-09 09:54:10.847581699 +0000 UTC m=+6.180065481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhtrb" (UniqueName: "kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb") pod "network-check-target-w8zqv" (UID: "2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:09.072467 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:09.072419 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 09:49:07 +0000 UTC" deadline="2028-01-05 20:44:21.28843894 +0000 UTC" Jul 09 09:54:09.072467 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:09.072465 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13090h50m12.215979174s" Jul 09 09:54:09.157272 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:09.157217 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:09.157445 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:09.157350 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:09.157809 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:09.157767 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:09.157905 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:09.157868 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:10.154663 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:10.154556 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:10.155105 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.154696 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:10.662864 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:10.662824 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:10.663055 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.662999 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:10.663117 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.663064 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs podName:49062adc-752f-4d87-aa95-db7f44411a17 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:14.663046352 +0000 UTC m=+9.995530131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs") pod "network-metrics-daemon-8kpn9" (UID: "49062adc-752f-4d87-aa95-db7f44411a17") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:10.764442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:10.763909 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:10.764442 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.764047 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:10.764442 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.764108 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret podName:048fd61f-ba5b-41bc-95a5-366a5734571d nodeName:}" failed. No retries permitted until 2026-07-09 09:54:14.764089536 +0000 UTC m=+10.096573336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret") pod "global-pull-secret-syncer-slmhl" (UID: "048fd61f-ba5b-41bc-95a5-366a5734571d") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:10.865138 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:10.864550 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:10.865138 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.864717 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:54:10.865138 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.864733 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:54:10.865138 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.864745 2571 projected.go:196] Error preparing data for projected volume kube-api-access-nhtrb for pod openshift-network-diagnostics/network-check-target-w8zqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:10.865138 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:10.864800 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb podName:2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf nodeName:}" failed. No retries permitted until 2026-07-09 09:54:14.864783856 +0000 UTC m=+10.197267635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhtrb" (UniqueName: "kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb") pod "network-check-target-w8zqv" (UID: "2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:11.156744 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:11.156238 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:11.156744 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:11.156353 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:11.157239 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:11.156800 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:11.157239 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:11.156906 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:12.154730 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:12.154652 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:12.154902 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:12.154803 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:13.154240 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:13.154190 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:13.154726 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:13.154313 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:13.154726 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:13.154189 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:13.154726 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:13.154499 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:14.154043 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:14.153970 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:14.154208 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.154098 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:14.694386 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:14.694345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:14.694838 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.694522 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:14.694838 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.694590 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs podName:49062adc-752f-4d87-aa95-db7f44411a17 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:22.694570827 +0000 UTC m=+18.027054606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs") pod "network-metrics-daemon-8kpn9" (UID: "49062adc-752f-4d87-aa95-db7f44411a17") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:14.795811 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:14.795734 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:14.795983 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.795903 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:14.795983 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.795964 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret podName:048fd61f-ba5b-41bc-95a5-366a5734571d nodeName:}" failed. No retries permitted until 2026-07-09 09:54:22.795948033 +0000 UTC m=+18.128431812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret") pod "global-pull-secret-syncer-slmhl" (UID: "048fd61f-ba5b-41bc-95a5-366a5734571d") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:14.896671 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:14.896620 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:14.896849 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.896777 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:54:14.896849 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.896801 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:54:14.896849 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.896813 2571 projected.go:196] Error preparing data for projected volume kube-api-access-nhtrb for pod openshift-network-diagnostics/network-check-target-w8zqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:14.896998 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:14.896875 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb podName:2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf nodeName:}" failed. No retries permitted until 2026-07-09 09:54:22.896856149 +0000 UTC m=+18.229339927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhtrb" (UniqueName: "kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb") pod "network-check-target-w8zqv" (UID: "2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:15.155830 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:15.155332 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:15.155830 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:15.155448 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:15.155830 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:15.155667 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:15.156097 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:15.155801 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:16.154515 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:16.154286 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:16.155038 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:16.154635 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:17.154600 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:17.154562 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:17.155053 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:17.154708 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:17.155053 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:17.154752 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:17.155053 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:17.154848 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:18.154135 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:18.154088 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:18.154269 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:18.154233 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:19.154025 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:19.153981 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:19.154466 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:19.154151 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:19.154466 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:19.154196 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:19.154466 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:19.154302 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:20.154325 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:20.154291 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:20.154734 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:20.154408 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:21.154305 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:21.154272 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:21.154484 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:21.154274 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:21.154484 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:21.154388 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:21.154484 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:21.154475 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:22.153889 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:22.153858 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:22.154088 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.153972 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:22.757170 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:22.757056 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:22.757588 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.757229 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:22.757588 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.757296 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs podName:49062adc-752f-4d87-aa95-db7f44411a17 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:38.757275802 +0000 UTC m=+34.089759590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs") pod "network-metrics-daemon-8kpn9" (UID: "49062adc-752f-4d87-aa95-db7f44411a17") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:22.857639 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:22.857600 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:22.857831 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.857732 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:22.857831 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.857793 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret podName:048fd61f-ba5b-41bc-95a5-366a5734571d nodeName:}" failed. No retries permitted until 2026-07-09 09:54:38.857774688 +0000 UTC m=+34.190258470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret") pod "global-pull-secret-syncer-slmhl" (UID: "048fd61f-ba5b-41bc-95a5-366a5734571d") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:22.958083 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:22.958051 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:22.958275 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.958230 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:54:22.958275 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.958252 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:54:22.958275 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.958265 2571 projected.go:196] Error preparing data for projected volume kube-api-access-nhtrb for pod openshift-network-diagnostics/network-check-target-w8zqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:22.958426 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:22.958326 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb podName:2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf nodeName:}" failed. No retries permitted until 2026-07-09 09:54:38.958309029 +0000 UTC m=+34.290792828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhtrb" (UniqueName: "kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb") pod "network-check-target-w8zqv" (UID: "2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:23.154935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:23.154857 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:23.155085 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:23.154987 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:23.155172 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:23.155075 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:23.155235 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:23.155202 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:24.154483 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:24.154450 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:24.154925 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:24.154575 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:25.155419 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:25.155381 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:25.155839 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:25.155498 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:25.155839 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:25.155548 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:25.155839 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:25.155663 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:26.154809 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:26.154781 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:26.154945 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:26.154881 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:27.154496 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.154335 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:27.155115 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.154429 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:27.155115 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:27.154599 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:27.155115 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:27.154649 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:27.221435 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.221401 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" event={"ID":"f5d7380ed3c9d0d9e6311c53297db640","Type":"ContainerStarted","Data":"7e621a263d71f1edb247cc2dad4977969706590dc31d4f5fc45f8b07c4ec9f20"} Jul 09 09:54:27.222944 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.222922 2571 generic.go:356] "Generic (PLEG): container finished" podID="6aacd596fa6102c9a13bb44707f0c6b4" containerID="be83b0d298e00006f25e076b925bf1c578c7653d156939426295efc45bfbedb8" exitCode=0 Jul 09 09:54:27.223034 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.222991 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" event={"ID":"6aacd596fa6102c9a13bb44707f0c6b4","Type":"ContainerDied","Data":"be83b0d298e00006f25e076b925bf1c578c7653d156939426295efc45bfbedb8"} Jul 09 09:54:27.224443 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.224416 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7xn2" event={"ID":"1b75e24a-dd42-4539-9577-707efd61daaa","Type":"ContainerStarted","Data":"fbf1f251b466a9c901077c5c6ef01af07b7a3e915ae5b1e021e29a60299ca5b7"} Jul 09 09:54:27.228806 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.226220 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tkpgz" event={"ID":"b5181675-9c76-4ae5-bb57-dd4c8558b80f","Type":"ContainerStarted","Data":"5d5f777087d30b7d02ea2461448e8925e66662d6aedc11b85fdcecd99f1fd469"} Jul 09 09:54:27.230407 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.230379 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k57ns" event={"ID":"8ff1e23f-f24b-4568-b9ee-1f0692a9afe3","Type":"ContainerStarted","Data":"52ef97a4ab5429d920c6c2338a2fa31c8de30c268118af406cb89abd281f6e80"} Jul 09 09:54:27.231910 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.231888 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" event={"ID":"13a3ff5c-d58e-4dcf-af73-5129ab6bbe8e","Type":"ContainerStarted","Data":"575e42822a81e04c64e84acc746db3cf681d1989eb05f97f2cb19c4aa6a712bb"} Jul 09 09:54:27.233380 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.233355 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" event={"ID":"97460b30-436a-49e4-9a03-b6ed61a25ca5","Type":"ContainerStarted","Data":"b06d48532707962aaf157e273a4a0ae4320f9f00ac83ef66b94b84bdb55d383b"} Jul 09 09:54:27.236247 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.236230 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"a54b9a4f8bf8be7ab1b7b73f62cfa0978c6d66d57937dad98a77744776c4e5c0"} Jul 09 09:54:27.236338 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.236291 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"c1968d41ef07a09357ec051cf04667f6b89b7c22410ddbf0e17840a0d2cefbd0"} Jul 09 09:54:27.236395 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.236337 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"c9c5bcaf629cd911ea4415fe3a700eaf429270eac5c09a364b5e3d74c24c8b6f"} Jul 09 09:54:27.236395 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.236377 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"3ac6fc8ea4a4f4f917bf32e2ab04bb2d9fba76bf2ccb2f3d19ff2ac05f1d5f6f"} Jul 09 09:54:27.236486 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.236406 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"f74c7caa0ed519737fb0b53daca0a0bb501f1e7e94f72717f8da6e3f855735f5"} Jul 09 09:54:27.236486 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.236441 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"537b2f13bbf38397a2083f7bc62d80f6c5d507d0942acba5c14141ae3b5188ee"} Jul 09 09:54:27.237744 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.237719 2571 generic.go:356] "Generic (PLEG): container finished" podID="1090cf01-53a3-486e-9641-90a2ca359480" containerID="469338ac7696edc9072bdacb771707950cedec58fba4010b4477d56042230450" exitCode=0 Jul 09 09:54:27.237846 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.237779 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" event={"ID":"1090cf01-53a3-486e-9641-90a2ca359480","Type":"ContainerDied","Data":"469338ac7696edc9072bdacb771707950cedec58fba4010b4477d56042230450"} Jul 09 09:54:27.239372 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.239308 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-69.ec2.internal" podStartSLOduration=21.239292854 podStartE2EDuration="21.239292854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:27.239045475 +0000 UTC m=+22.571529274" watchObservedRunningTime="2026-07-09 09:54:27.239292854 +0000 UTC m=+22.571776654" Jul 09 09:54:27.240024 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.240000 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l84jq" event={"ID":"d31879ac-51f8-4f04-8591-dd2c0487c68b","Type":"ContainerStarted","Data":"16e049b1ba48ed4ee0dff1720a2a68ea97432a7040bae89f151cde7e504e68c4"} Jul 09 09:54:27.258137 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.258092 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dm2pn" podStartSLOduration=3.5688873450000003 podStartE2EDuration="22.258080363s" totalImagesPullingTime="18.689193018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:27.257296068 +0000 UTC m=+22.589779866" watchObservedRunningTime="2026-07-09 09:54:27.258080363 +0000 UTC m=+22.590564161" Jul 09 09:54:27.279523 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.279488 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k57ns" podStartSLOduration=7.259662432 podStartE2EDuration="22.279476386s" totalImagesPullingTime="15.019813954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:27.278809344 +0000 UTC m=+22.611293141" watchObservedRunningTime="2026-07-09 09:54:27.279476386 +0000 UTC m=+22.611960188" Jul 09 09:54:27.347746 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.347702 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w7xn2" podStartSLOduration=3.431623074 podStartE2EDuration="22.347688226s" totalImagesPullingTime="18.916065152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:27.330406396 +0000 UTC m=+22.662890193" watchObservedRunningTime="2026-07-09 09:54:27.347688226 +0000 UTC m=+22.680172024" Jul 09 09:54:27.347893 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.347822 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tkpgz" podStartSLOduration=7.334991208 podStartE2EDuration="22.347816244s" totalImagesPullingTime="15.012825036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:27.347371741 +0000 UTC m=+22.679855541" watchObservedRunningTime="2026-07-09 09:54:27.347816244 +0000 UTC m=+22.680300042" Jul 09 09:54:27.364833 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:27.364793 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l84jq" podStartSLOduration=3.4585405590000002 podStartE2EDuration="22.364780603s" totalImagesPullingTime="18.906240044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:27.36471894 +0000 UTC m=+22.697202738" watchObservedRunningTime="2026-07-09 09:54:27.364780603 +0000 UTC m=+22.697264428" Jul 09 09:54:28.154229 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:28.154191 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:28.154486 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:28.154311 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:28.243442 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:28.243396 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9rbg7" event={"ID":"08c3e090-2c2d-4552-8d3b-5fb7c7cfc4b6","Type":"ContainerStarted","Data":"fc95cfb46fae17a3e4eeed14d607af5967c21dd2205d6348d275b7c51c8843df"} Jul 09 09:54:28.278068 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:28.278013 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9rbg7" podStartSLOduration=8.214202052 podStartE2EDuration="23.277992291s" totalImagesPullingTime="15.063790239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:28.260610549 +0000 UTC m=+23.593094348" watchObservedRunningTime="2026-07-09 09:54:28.277992291 +0000 UTC m=+23.610476089" Jul 09 09:54:28.742177 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:28.742147 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 09:54:29.079346 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:29.079186 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T09:54:28.742171534Z","UUID":"4e3f9656-1cfb-46b0-a877-665e6566caac","Handler":null,"Name":"","Endpoint":""} Jul 09 09:54:29.080670 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:29.080648 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 09:54:29.080788 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:29.080690 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 09:54:29.154342 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:29.154313 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:29.154342 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:29.154352 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:29.154569 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:29.154439 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:29.154569 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:29.154522 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:29.247106 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:29.247078 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" event={"ID":"6aacd596fa6102c9a13bb44707f0c6b4","Type":"ContainerStarted","Data":"36514e0e6d99d14d1de1e615a9ed66feff872ba6c078a9b10bc39ac52151e807"} Jul 09 09:54:29.248662 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:29.248641 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" event={"ID":"97460b30-436a-49e4-9a03-b6ed61a25ca5","Type":"ContainerStarted","Data":"6198c7de9cb46453d4c9c60ce936f94a7c89aac0571fa72deb5f0db385f4ae78"} Jul 09 09:54:29.277938 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:29.277899 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-69.ec2.internal" podStartSLOduration=23.277885696 podStartE2EDuration="23.277885696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:29.277283532 +0000 UTC m=+24.609767330" watchObservedRunningTime="2026-07-09 09:54:29.277885696 +0000 UTC m=+24.610369497" Jul 09 09:54:30.138732 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:30.138657 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:30.139225 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:30.139211 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:30.154291 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:30.154262 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:30.154428 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:30.154410 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:30.253422 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:30.253389 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" event={"ID":"97460b30-436a-49e4-9a03-b6ed61a25ca5","Type":"ContainerStarted","Data":"0bb1309972d73fe1fa85a561fd25686c2ddb9a0dce6a493fd331f9aa31cfe9ae"} Jul 09 09:54:30.256592 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:30.256567 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"5667277c7166975521d1d02d9ff93a4a587a9c104c73c092e16d45a2b41232d8"} Jul 09 09:54:30.256706 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:30.256633 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:30.257107 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:30.257092 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tkpgz" Jul 09 09:54:30.271915 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:30.271874 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t4mkh" podStartSLOduration=3.7117043709999997 podStartE2EDuration="25.271858565s" totalImagesPullingTime="21.560154194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:30.271546001 +0000 UTC m=+25.604029809" watchObservedRunningTime="2026-07-09 09:54:30.271858565 +0000 UTC m=+25.604342363" Jul 09 09:54:31.154587 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:31.154553 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:31.154587 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:31.154577 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:31.154820 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:31.154681 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:31.154820 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:31.154778 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:32.154664 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:32.154632 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:32.155067 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:32.154753 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:33.154734 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.154569 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:33.155483 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.154569 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:33.155483 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:33.154834 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:33.155483 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:33.154892 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:33.266245 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.266161 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" event={"ID":"d41af354-3b0c-4774-98df-f67897540b38","Type":"ContainerStarted","Data":"1ffd5ce36b957c4e61e07ae7ed8d4c10c9c83e2ca5afe726ba0215d8dbea728f"} Jul 09 09:54:33.266407 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.266387 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:33.266477 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.266461 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:33.268014 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.267988 2571 generic.go:356] "Generic (PLEG): container finished" podID="1090cf01-53a3-486e-9641-90a2ca359480" containerID="db7cc171edc743492055da564c39cbb4c6906e5ae971e8c944d8fc35eb918a5d" exitCode=0 Jul 09 09:54:33.268237 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.268025 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" event={"ID":"1090cf01-53a3-486e-9641-90a2ca359480","Type":"ContainerDied","Data":"db7cc171edc743492055da564c39cbb4c6906e5ae971e8c944d8fc35eb918a5d"} Jul 09 09:54:33.281435 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.281414 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:33.295549 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:33.295505 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" podStartSLOduration=9.090861189 podStartE2EDuration="28.295491323s" totalImagesPullingTime="19.204630134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:33.29475343 +0000 UTC m=+28.627237229" watchObservedRunningTime="2026-07-09 09:54:33.295491323 +0000 UTC m=+28.627975120" Jul 09 09:54:34.154402 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.154324 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:34.154538 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:34.154442 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:34.270662 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.270633 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:34.285484 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.285458 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:54:34.793071 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.792991 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-slmhl"] Jul 09 09:54:34.793254 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.793120 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:34.793320 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:34.793270 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:34.795952 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.795928 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w8zqv"] Jul 09 09:54:34.796071 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.796013 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:34.796120 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:34.796081 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:34.798542 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.798520 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8kpn9"] Jul 09 09:54:34.798622 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:34.798610 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:34.798741 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:34.798713 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:35.274290 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:35.274210 2571 generic.go:356] "Generic (PLEG): container finished" podID="1090cf01-53a3-486e-9641-90a2ca359480" containerID="c8532f04e738a6bca81839834388d6a891c3fd146cfa7c48101f2e64e2e727d5" exitCode=0 Jul 09 09:54:35.274615 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:35.274300 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" event={"ID":"1090cf01-53a3-486e-9641-90a2ca359480","Type":"ContainerDied","Data":"c8532f04e738a6bca81839834388d6a891c3fd146cfa7c48101f2e64e2e727d5"} Jul 09 09:54:36.154858 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:36.154771 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:36.154858 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:36.154803 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:36.154858 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:36.154815 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:36.155118 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:36.154898 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:36.155118 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:36.155019 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:36.155118 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:36.155101 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:37.280935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:37.280852 2571 generic.go:356] "Generic (PLEG): container finished" podID="1090cf01-53a3-486e-9641-90a2ca359480" containerID="9120ee308e2ef9ea0ce0d67ec281485bd008af1d8ddfae2097d82c9f626a3719" exitCode=0 Jul 09 09:54:37.280935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:37.280894 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" event={"ID":"1090cf01-53a3-486e-9641-90a2ca359480","Type":"ContainerDied","Data":"9120ee308e2ef9ea0ce0d67ec281485bd008af1d8ddfae2097d82c9f626a3719"} Jul 09 09:54:38.154347 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:38.154318 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:38.154518 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:38.154414 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:38.154518 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:38.154423 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:38.154518 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.154505 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8zqv" podUID="2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf" Jul 09 09:54:38.154619 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.154567 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8kpn9" podUID="49062adc-752f-4d87-aa95-db7f44411a17" Jul 09 09:54:38.154619 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.154418 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-slmhl" podUID="048fd61f-ba5b-41bc-95a5-366a5734571d" Jul 09 09:54:38.780310 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:38.780272 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:38.780706 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.780443 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:38.780706 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.780523 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs podName:49062adc-752f-4d87-aa95-db7f44411a17 nodeName:}" failed. No retries permitted until 2026-07-09 09:55:10.7805026 +0000 UTC m=+66.112986381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs") pod "network-metrics-daemon-8kpn9" (UID: "49062adc-752f-4d87-aa95-db7f44411a17") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 09:54:38.880923 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:38.880889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:38.881075 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.881054 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:38.881155 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.881142 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret podName:048fd61f-ba5b-41bc-95a5-366a5734571d nodeName:}" failed. No retries permitted until 2026-07-09 09:55:10.881107124 +0000 UTC m=+66.213590920 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret") pod "global-pull-secret-syncer-slmhl" (UID: "048fd61f-ba5b-41bc-95a5-366a5734571d") : object "kube-system"/"original-pull-secret" not registered Jul 09 09:54:38.982252 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:38.982221 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:38.982413 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.982363 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 09:54:38.982413 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.982384 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 09:54:38.982413 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.982394 2571 projected.go:196] Error preparing data for projected volume kube-api-access-nhtrb for pod openshift-network-diagnostics/network-check-target-w8zqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:38.982528 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:38.982447 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb podName:2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf nodeName:}" failed. No retries permitted until 2026-07-09 09:55:10.982433976 +0000 UTC m=+66.314917752 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhtrb" (UniqueName: "kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb") pod "network-check-target-w8zqv" (UID: "2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 09:54:39.421598 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.421568 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-69.ec2.internal" event="NodeReady" Jul 09 09:54:39.421845 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.421701 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 09:54:39.461344 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.461317 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78f9ddf4fc-2m6lk"] Jul 09 09:54:39.463835 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.463803 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.466685 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.466467 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 09:54:39.466803 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.466715 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jpdnh\"" Jul 09 09:54:39.467021 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.467004 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 09:54:39.467275 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.467255 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 09:54:39.472820 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.472800 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 09:54:39.474763 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.474703 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wp9sj"] Jul 09 09:54:39.477622 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.477605 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6zbgj"] Jul 09 09:54:39.477760 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.477743 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.480434 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.480408 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6pb2\"" Jul 09 09:54:39.480655 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.480634 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6zbgj" Jul 09 09:54:39.480772 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.480753 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 09:54:39.481210 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.481035 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f9ddf4fc-2m6lk"] Jul 09 09:54:39.481210 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.481047 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 09:54:39.483103 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.483081 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 09:54:39.483528 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.483356 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 09:54:39.483528 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.483364 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 09:54:39.483528 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.483405 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-df4wp\"" Jul 09 09:54:39.487037 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.487018 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6zbgj"] Jul 09 09:54:39.490331 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.490311 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wp9sj"] Jul 09 09:54:39.586509 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586471 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-bound-sa-token\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.586666 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586531 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-certificates\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.586666 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586589 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqpc\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-kube-api-access-rlqpc\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.586666 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586629 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-669c8\" (UniqueName: \"kubernetes.io/projected/5b882c2a-8f11-4f25-aeb0-da5a38087884-kube-api-access-669c8\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.586666 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkp9\" (UniqueName: \"kubernetes.io/projected/de11cbd5-8899-4f7f-b5db-e2236850cb4c-kube-api-access-zjkp9\") pod \"ingress-canary-6zbgj\" (UID: \"de11cbd5-8899-4f7f-b5db-e2236850cb4c\") " pod="openshift-ingress-canary/ingress-canary-6zbgj" Jul 09 09:54:39.586837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-ca-trust-extracted\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.586837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586722 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b882c2a-8f11-4f25-aeb0-da5a38087884-config-volume\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.586837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586755 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.586837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586784 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b882c2a-8f11-4f25-aeb0-da5a38087884-tmp-dir\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.586837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586811 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.586837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586827 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-trusted-ca\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.587073 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586877 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-installation-pull-secrets\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.587073 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586897 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de11cbd5-8899-4f7f-b5db-e2236850cb4c-cert\") pod \"ingress-canary-6zbgj\" (UID: \"de11cbd5-8899-4f7f-b5db-e2236850cb4c\") " pod="openshift-ingress-canary/ingress-canary-6zbgj" Jul 09 09:54:39.587073 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.586927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-image-registry-private-configuration\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.659064 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.659022 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tkpgz_b5181675-9c76-4ae5-bb57-dd4c8558b80f/konnectivity-agent/0.log" limit=1024 Jul 09 09:54:39.687564 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687486 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de11cbd5-8899-4f7f-b5db-e2236850cb4c-cert\") pod \"ingress-canary-6zbgj\" (UID: \"de11cbd5-8899-4f7f-b5db-e2236850cb4c\") " pod="openshift-ingress-canary/ingress-canary-6zbgj" Jul 09 09:54:39.687564 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687526 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-image-registry-private-configuration\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.687564 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-bound-sa-token\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.687810 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687574 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-certificates\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.687810 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687599 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqpc\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-kube-api-access-rlqpc\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.687810 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687727 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-669c8\" (UniqueName: \"kubernetes.io/projected/5b882c2a-8f11-4f25-aeb0-da5a38087884-kube-api-access-669c8\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.687810 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkp9\" (UniqueName: \"kubernetes.io/projected/de11cbd5-8899-4f7f-b5db-e2236850cb4c-kube-api-access-zjkp9\") pod \"ingress-canary-6zbgj\" (UID: \"de11cbd5-8899-4f7f-b5db-e2236850cb4c\") " pod="openshift-ingress-canary/ingress-canary-6zbgj" Jul 09 09:54:39.687991 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687825 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-ca-trust-extracted\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.687991 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687851 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b882c2a-8f11-4f25-aeb0-da5a38087884-config-volume\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.687991 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.687991 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b882c2a-8f11-4f25-aeb0-da5a38087884-tmp-dir\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.687991 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687934 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.687991 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.687954 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-trusted-ca\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.688266 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.688016 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-installation-pull-secrets\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.688654 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:39.688374 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 09:54:39.688654 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:39.688396 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9ddf4fc-2m6lk: secret "image-registry-tls" not found Jul 09 09:54:39.688654 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:39.688465 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls podName:57cc59ae-e2bf-4fde-86f2-e1878ae23bb8 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:40.188445208 +0000 UTC m=+35.520928990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls") pod "image-registry-78f9ddf4fc-2m6lk" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8") : secret "image-registry-tls" not found Jul 09 09:54:39.688654 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.688528 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b882c2a-8f11-4f25-aeb0-da5a38087884-tmp-dir\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.688654 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.688579 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-certificates\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.688654 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.688615 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-ca-trust-extracted\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.688654 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:39.688638 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 09:54:39.689055 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:39.688696 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls podName:5b882c2a-8f11-4f25-aeb0-da5a38087884 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:40.18868126 +0000 UTC m=+35.521165038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls") pod "dns-default-wp9sj" (UID: "5b882c2a-8f11-4f25-aeb0-da5a38087884") : secret "dns-default-metrics-tls" not found Jul 09 09:54:39.689055 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.688779 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b882c2a-8f11-4f25-aeb0-da5a38087884-config-volume\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.689536 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.689505 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-trusted-ca\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.692231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.692117 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de11cbd5-8899-4f7f-b5db-e2236850cb4c-cert\") pod \"ingress-canary-6zbgj\" (UID: \"de11cbd5-8899-4f7f-b5db-e2236850cb4c\") " pod="openshift-ingress-canary/ingress-canary-6zbgj" Jul 09 09:54:39.692333 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.692163 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-image-registry-private-configuration\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.692333 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.692182 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-installation-pull-secrets\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.697801 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.697757 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-669c8\" (UniqueName: \"kubernetes.io/projected/5b882c2a-8f11-4f25-aeb0-da5a38087884-kube-api-access-669c8\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:39.698066 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.698043 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqpc\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-kube-api-access-rlqpc\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.698375 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.698342 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-bound-sa-token\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:39.698892 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.698836 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkp9\" (UniqueName: \"kubernetes.io/projected/de11cbd5-8899-4f7f-b5db-e2236850cb4c-kube-api-access-zjkp9\") pod \"ingress-canary-6zbgj\" (UID: \"de11cbd5-8899-4f7f-b5db-e2236850cb4c\") " pod="openshift-ingress-canary/ingress-canary-6zbgj" Jul 09 09:54:39.797098 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.797061 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6zbgj" Jul 09 09:54:39.820477 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.820450 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tkpgz_b5181675-9c76-4ae5-bb57-dd4c8558b80f/konnectivity-agent/0.log" limit=1024 Jul 09 09:54:39.955218 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:39.952908 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6zbgj"] Jul 09 09:54:39.956684 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:39.956651 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde11cbd5_8899_4f7f_b5db_e2236850cb4c.slice/crio-83efca17faf6f0e9f12fd1d111dcb0e51add6bacf2797a29f87377b4975c96de WatchSource:0}: Error finding container 83efca17faf6f0e9f12fd1d111dcb0e51add6bacf2797a29f87377b4975c96de: Status 404 returned error can't find the container with id 83efca17faf6f0e9f12fd1d111dcb0e51add6bacf2797a29f87377b4975c96de Jul 09 09:54:40.154672 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.154642 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:54:40.154837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.154755 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:54:40.154837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.154796 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:54:40.158013 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.157975 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 09:54:40.158171 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.158028 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 09:54:40.158171 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.158041 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 09:54:40.158171 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.158066 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4794j\"" Jul 09 09:54:40.158171 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.158115 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffmqr\"" Jul 09 09:54:40.158171 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.158066 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 09:54:40.192353 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.192329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:40.192476 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.192360 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:40.192476 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:40.192473 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 09:54:40.192588 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:40.192481 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 09:54:40.192588 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:40.192555 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls podName:5b882c2a-8f11-4f25-aeb0-da5a38087884 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:41.192534642 +0000 UTC m=+36.525018424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls") pod "dns-default-wp9sj" (UID: "5b882c2a-8f11-4f25-aeb0-da5a38087884") : secret "dns-default-metrics-tls" not found Jul 09 09:54:40.192588 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:40.192484 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9ddf4fc-2m6lk: secret "image-registry-tls" not found Jul 09 09:54:40.192733 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:40.192604 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls podName:57cc59ae-e2bf-4fde-86f2-e1878ae23bb8 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:41.192592911 +0000 UTC m=+36.525076687 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls") pod "image-registry-78f9ddf4fc-2m6lk" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8") : secret "image-registry-tls" not found Jul 09 09:54:40.288899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.288817 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6zbgj" event={"ID":"de11cbd5-8899-4f7f-b5db-e2236850cb4c","Type":"ContainerStarted","Data":"83efca17faf6f0e9f12fd1d111dcb0e51add6bacf2797a29f87377b4975c96de"} Jul 09 09:54:40.388248 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.388153 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tkpgz_b5181675-9c76-4ae5-bb57-dd4c8558b80f/konnectivity-agent/0.log" limit=1024 Jul 09 09:54:40.867642 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.867605 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-fpvm6"] Jul 09 09:54:40.886698 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.886666 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-fpvm6"] Jul 09 09:54:40.886851 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.886772 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:40.890628 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.890400 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 09:54:40.890628 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.890436 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 09:54:40.890628 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.890444 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 09:54:40.890937 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.890678 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 09:54:40.892341 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.892323 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nfs9s\"" Jul 09 09:54:40.898156 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.898088 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 09:54:40.968193 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.968157 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp"] Jul 09 09:54:40.993765 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.993738 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498"] Jul 09 09:54:40.993920 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.993848 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:40.996830 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.996800 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:54:40.996830 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.996800 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 09:54:40.997005 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.996807 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xjjkj\"" Jul 09 09:54:40.997005 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.996896 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 09:54:40.997694 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.997674 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-serving-cert\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:40.997777 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.997732 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-snapshots\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:40.997821 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.997793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-tmp\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:40.997821 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.997811 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:40.997897 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.997826 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69pxz\" (UniqueName: \"kubernetes.io/projected/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-kube-api-access-69pxz\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:40.997897 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:40.997846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.016743 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.016721 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml"] Jul 09 09:54:41.016854 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.016837 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498" Jul 09 09:54:41.019780 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.019672 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4pdz2\"" Jul 09 09:54:41.019780 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.019672 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:54:41.019780 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.019729 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 09:54:41.036887 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.036863 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56f6dfc6c5-m7gq8"] Jul 09 09:54:41.037017 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.036993 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.039911 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.039882 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 09:54:41.040101 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.039970 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dstrq\"" Jul 09 09:54:41.040208 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.040190 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 09 09:54:41.044554 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.041540 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 09:54:41.044554 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.044441 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 09 09:54:41.053505 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.053484 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp"] Jul 09 09:54:41.053597 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.053528 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498"] Jul 09 09:54:41.053597 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.053551 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml"] Jul 09 09:54:41.053597 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.053570 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56f6dfc6c5-m7gq8"] Jul 09 09:54:41.053597 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.053593 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.056306 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.056279 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 09:54:41.056306 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.056298 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 09:54:41.056306 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.056308 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 09:54:41.056514 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.056337 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 09:54:41.056514 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.056290 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tqxqm\"" Jul 09 09:54:41.056514 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.056443 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 09:54:41.056514 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.056472 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 09:54:41.098697 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8gjz\" (UniqueName: \"kubernetes.io/projected/bee94409-6c0f-47f2-8107-f0e93537ce33-kube-api-access-g8gjz\") pod \"volume-data-source-validator-5d456fdbcd-fb498\" (UID: \"bee94409-6c0f-47f2-8107-f0e93537ce33\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498" Jul 09 09:54:41.098813 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098699 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-466t4\" (UniqueName: \"kubernetes.io/projected/88eeac34-fbf1-41c0-956e-7ee749407b8a-kube-api-access-466t4\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:41.098813 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-snapshots\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.098945 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098832 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-tmp\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.098945 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.098945 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098883 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69pxz\" (UniqueName: \"kubernetes.io/projected/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-kube-api-access-69pxz\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.098945 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098915 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.099180 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098955 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:41.099180 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.098984 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-serving-cert\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.099589 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.099564 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-tmp\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.099729 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.099642 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.099729 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.099721 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-snapshots\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.099889 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.099827 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.102341 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.102320 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-serving-cert\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.109682 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.109657 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69pxz\" (UniqueName: \"kubernetes.io/projected/90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4-kube-api-access-69pxz\") pod \"insights-operator-5c847dcfc4-fpvm6\" (UID: \"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4\") " pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.198726 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.198646 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" Jul 09 09:54:41.199959 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.199931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.200073 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.199974 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhsl\" (UniqueName: \"kubernetes.io/projected/35d1d4bd-3fef-4335-a2b3-8085386414da-kube-api-access-5mhsl\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.200073 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200013 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:41.200213 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200081 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-default-certificate\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.200213 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.200112 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 09:54:41.200213 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/35d1d4bd-3fef-4335-a2b3-8085386414da-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.200213 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.200194 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls podName:5b882c2a-8f11-4f25-aeb0-da5a38087884 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:43.200175545 +0000 UTC m=+38.532659324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls") pod "dns-default-wp9sj" (UID: "5b882c2a-8f11-4f25-aeb0-da5a38087884") : secret "dns-default-metrics-tls" not found Jul 09 09:54:41.200414 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200216 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:41.200414 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.200302 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 09:54:41.200414 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.200318 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9ddf4fc-2m6lk: secret "image-registry-tls" not found Jul 09 09:54:41.200414 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200324 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:41.200414 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.200414 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.200392 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls podName:57cc59ae-e2bf-4fde-86f2-e1878ae23bb8 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:43.200375805 +0000 UTC m=+38.532859595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls") pod "image-registry-78f9ddf4fc-2m6lk" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8") : secret "image-registry-tls" not found Jul 09 09:54:41.200717 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.200454 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 09:54:41.200717 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.200512 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls podName:88eeac34-fbf1-41c0-956e-7ee749407b8a nodeName:}" failed. No retries permitted until 2026-07-09 09:54:41.700497145 +0000 UTC m=+37.032980939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nsqsp" (UID: "88eeac34-fbf1-41c0-956e-7ee749407b8a") : secret "samples-operator-tls" not found Jul 09 09:54:41.200717 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.200717 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200585 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-stats-auth\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.200717 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200628 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8gjz\" (UniqueName: \"kubernetes.io/projected/bee94409-6c0f-47f2-8107-f0e93537ce33-kube-api-access-g8gjz\") pod \"volume-data-source-validator-5d456fdbcd-fb498\" (UID: \"bee94409-6c0f-47f2-8107-f0e93537ce33\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498" Jul 09 09:54:41.200717 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200658 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-466t4\" (UniqueName: \"kubernetes.io/projected/88eeac34-fbf1-41c0-956e-7ee749407b8a-kube-api-access-466t4\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:41.200717 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.200682 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smt95\" (UniqueName: \"kubernetes.io/projected/ed295098-19e7-483e-88c5-20da6182ea79-kube-api-access-smt95\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.210708 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.210678 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-466t4\" (UniqueName: \"kubernetes.io/projected/88eeac34-fbf1-41c0-956e-7ee749407b8a-kube-api-access-466t4\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:41.227514 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.227459 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8gjz\" (UniqueName: \"kubernetes.io/projected/bee94409-6c0f-47f2-8107-f0e93537ce33-kube-api-access-g8gjz\") pod \"volume-data-source-validator-5d456fdbcd-fb498\" (UID: \"bee94409-6c0f-47f2-8107-f0e93537ce33\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498" Jul 09 09:54:41.251832 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.251767 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tkpgz_b5181675-9c76-4ae5-bb57-dd4c8558b80f/konnectivity-agent/0.log" limit=1024 Jul 09 09:54:41.302357 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.302311 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-stats-auth\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.302483 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.302382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smt95\" (UniqueName: \"kubernetes.io/projected/ed295098-19e7-483e-88c5-20da6182ea79-kube-api-access-smt95\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.302483 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.302438 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.302483 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.302462 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhsl\" (UniqueName: \"kubernetes.io/projected/35d1d4bd-3fef-4335-a2b3-8085386414da-kube-api-access-5mhsl\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.302581 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.302505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-default-certificate\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.302581 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.302551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/35d1d4bd-3fef-4335-a2b3-8085386414da-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.302641 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.302607 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.302675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.302660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.302838 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.302821 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:41.802799693 +0000 UTC m=+37.135283490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : configmap references non-existent config key: service-ca.crt Jul 09 09:54:41.303565 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.303546 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 09:54:41.303667 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.303607 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:41.803590032 +0000 UTC m=+37.136073829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : secret "router-metrics-certs-default" not found Jul 09 09:54:41.304478 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.304454 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/35d1d4bd-3fef-4335-a2b3-8085386414da-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.304791 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.304773 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:41.305024 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.305008 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls podName:35d1d4bd-3fef-4335-a2b3-8085386414da nodeName:}" failed. No retries permitted until 2026-07-09 09:54:41.804980814 +0000 UTC m=+37.137464597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-q7kml" (UID: "35d1d4bd-3fef-4335-a2b3-8085386414da") : secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:41.306512 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.306488 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-default-certificate\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.307533 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.307500 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-stats-auth\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.315403 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.315358 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhsl\" (UniqueName: \"kubernetes.io/projected/35d1d4bd-3fef-4335-a2b3-8085386414da-kube-api-access-5mhsl\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.316016 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.315977 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smt95\" (UniqueName: \"kubernetes.io/projected/ed295098-19e7-483e-88c5-20da6182ea79-kube-api-access-smt95\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.326959 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.326756 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498" Jul 09 09:54:41.352236 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.352169 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tkpgz_b5181675-9c76-4ae5-bb57-dd4c8558b80f/konnectivity-agent/0.log" limit=1024 Jul 09 09:54:41.368070 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.367934 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-fpvm6"] Jul 09 09:54:41.371203 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:41.371175 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c650b7_c9b1_43a5_b6a4_4c0a9d68f7b4.slice/crio-448e363b2b71ac2902ee42dc26c302b63ef5bb1c6f88e11bc11ad2f0847bf017 WatchSource:0}: Error finding container 448e363b2b71ac2902ee42dc26c302b63ef5bb1c6f88e11bc11ad2f0847bf017: Status 404 returned error can't find the container with id 448e363b2b71ac2902ee42dc26c302b63ef5bb1c6f88e11bc11ad2f0847bf017 Jul 09 09:54:41.480613 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.480347 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498"] Jul 09 09:54:41.706707 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.706660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:41.706884 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.706822 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 09:54:41.706972 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.706914 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls podName:88eeac34-fbf1-41c0-956e-7ee749407b8a nodeName:}" failed. No retries permitted until 2026-07-09 09:54:42.706890668 +0000 UTC m=+38.039374461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nsqsp" (UID: "88eeac34-fbf1-41c0-956e-7ee749407b8a") : secret "samples-operator-tls" not found Jul 09 09:54:41.808233 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.808147 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.808233 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.808213 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:41.808461 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.808278 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:41.808461 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.808335 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 09:54:41.808461 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.808377 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:42.808357112 +0000 UTC m=+38.140840888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : configmap references non-existent config key: service-ca.crt Jul 09 09:54:41.808461 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.808384 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:41.808461 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.808403 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:42.80839347 +0000 UTC m=+38.140877247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : secret "router-metrics-certs-default" not found Jul 09 09:54:41.808461 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:41.808423 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls podName:35d1d4bd-3fef-4335-a2b3-8085386414da nodeName:}" failed. No retries permitted until 2026-07-09 09:54:42.808411896 +0000 UTC m=+38.140895671 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-q7kml" (UID: "35d1d4bd-3fef-4335-a2b3-8085386414da") : secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:41.863388 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.863355 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-xph5s"] Jul 09 09:54:41.880670 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.880636 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-xph5s"] Jul 09 09:54:41.881114 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.880762 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:41.883459 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.883410 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 09:54:41.883692 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.883674 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 09:54:41.883812 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.883717 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 09:54:41.883812 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.883800 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qkn5q\"" Jul 09 09:54:41.883991 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.883965 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:54:41.889450 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:41.889432 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 09:54:42.010088 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.010052 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-trusted-ca\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.010279 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.010246 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-config\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.010279 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.010277 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-serving-cert\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.010386 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.010295 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rnmz\" (UniqueName: \"kubernetes.io/projected/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-kube-api-access-5rnmz\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.111748 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.111710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-config\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.111896 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.111754 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-serving-cert\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.111896 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.111808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rnmz\" (UniqueName: \"kubernetes.io/projected/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-kube-api-access-5rnmz\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.112004 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.111898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-trusted-ca\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.112605 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.112575 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-config\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.112847 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.112801 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-trusted-ca\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.115698 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.115660 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-serving-cert\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.125497 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.125476 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rnmz\" (UniqueName: \"kubernetes.io/projected/48bad99c-20b4-4c34-b8eb-67d0e7189c8c-kube-api-access-5rnmz\") pod \"console-operator-68b757fb67-xph5s\" (UID: \"48bad99c-20b4-4c34-b8eb-67d0e7189c8c\") " pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.192820 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.192784 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:42.296205 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.296168 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" event={"ID":"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4","Type":"ContainerStarted","Data":"448e363b2b71ac2902ee42dc26c302b63ef5bb1c6f88e11bc11ad2f0847bf017"} Jul 09 09:54:42.718266 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.718228 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:42.718449 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:42.718405 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 09:54:42.718514 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:42.718479 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls podName:88eeac34-fbf1-41c0-956e-7ee749407b8a nodeName:}" failed. No retries permitted until 2026-07-09 09:54:44.718458447 +0000 UTC m=+40.050942226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nsqsp" (UID: "88eeac34-fbf1-41c0-956e-7ee749407b8a") : secret "samples-operator-tls" not found Jul 09 09:54:42.819073 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.819039 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:42.819241 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.819092 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:42.819241 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:42.819168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:42.819241 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:42.819230 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 09:54:42.819403 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:42.819263 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:42.819403 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:42.819286 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:44.819265572 +0000 UTC m=+40.151749368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : configmap references non-existent config key: service-ca.crt Jul 09 09:54:42.819403 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:42.819314 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:44.819303321 +0000 UTC m=+40.151787101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : secret "router-metrics-certs-default" not found Jul 09 09:54:42.819403 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:42.819328 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls podName:35d1d4bd-3fef-4335-a2b3-8085386414da nodeName:}" failed. No retries permitted until 2026-07-09 09:54:44.819320176 +0000 UTC m=+40.151803958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-q7kml" (UID: "35d1d4bd-3fef-4335-a2b3-8085386414da") : secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:42.987033 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:42.986959 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee94409_6c0f_47f2_8107_f0e93537ce33.slice/crio-914e84bc099c7ac8012e02fcd062fdfc20638a34146fe96136786a59915cab80 WatchSource:0}: Error finding container 914e84bc099c7ac8012e02fcd062fdfc20638a34146fe96136786a59915cab80: Status 404 returned error can't find the container with id 914e84bc099c7ac8012e02fcd062fdfc20638a34146fe96136786a59915cab80 Jul 09 09:54:43.224501 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:43.224459 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:43.224682 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:43.224595 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:43.224682 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:43.224620 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 09:54:43.224682 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:43.224639 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9ddf4fc-2m6lk: secret "image-registry-tls" not found Jul 09 09:54:43.224839 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:43.224691 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 09:54:43.224839 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:43.224697 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls podName:57cc59ae-e2bf-4fde-86f2-e1878ae23bb8 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:47.224682624 +0000 UTC m=+42.557166400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls") pod "image-registry-78f9ddf4fc-2m6lk" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8") : secret "image-registry-tls" not found Jul 09 09:54:43.224839 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:43.224755 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls podName:5b882c2a-8f11-4f25-aeb0-da5a38087884 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:47.224738803 +0000 UTC m=+42.557222581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls") pod "dns-default-wp9sj" (UID: "5b882c2a-8f11-4f25-aeb0-da5a38087884") : secret "dns-default-metrics-tls" not found Jul 09 09:54:43.300004 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:43.299908 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498" event={"ID":"bee94409-6c0f-47f2-8107-f0e93537ce33","Type":"ContainerStarted","Data":"914e84bc099c7ac8012e02fcd062fdfc20638a34146fe96136786a59915cab80"} Jul 09 09:54:44.740487 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:44.740453 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:44.740892 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:44.740601 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 09:54:44.740892 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:44.740673 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls podName:88eeac34-fbf1-41c0-956e-7ee749407b8a nodeName:}" failed. No retries permitted until 2026-07-09 09:54:48.740655876 +0000 UTC m=+44.073139656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nsqsp" (UID: "88eeac34-fbf1-41c0-956e-7ee749407b8a") : secret "samples-operator-tls" not found Jul 09 09:54:44.841219 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:44.841191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:44.841316 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:44.841299 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:44.841380 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:44.841329 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:44.841380 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:44.841353 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:44.841474 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:44.841400 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls podName:35d1d4bd-3fef-4335-a2b3-8085386414da nodeName:}" failed. No retries permitted until 2026-07-09 09:54:48.841376325 +0000 UTC m=+44.173860122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-q7kml" (UID: "35d1d4bd-3fef-4335-a2b3-8085386414da") : secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:44.841474 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:44.841439 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 09:54:44.841566 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:44.841476 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:48.841459977 +0000 UTC m=+44.173943756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : configmap references non-existent config key: service-ca.crt Jul 09 09:54:44.841566 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:44.841498 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:48.841486826 +0000 UTC m=+44.173970601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : secret "router-metrics-certs-default" not found Jul 09 09:54:44.962304 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:44.962195 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-xph5s"] Jul 09 09:54:44.966311 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:44.966289 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bad99c_20b4_4c34_b8eb_67d0e7189c8c.slice/crio-7c7a99c0733e6ff25ca7e465ed8aec9926cd5b05fadd5c71dc45071f897d6abe WatchSource:0}: Error finding container 7c7a99c0733e6ff25ca7e465ed8aec9926cd5b05fadd5c71dc45071f897d6abe: Status 404 returned error can't find the container with id 7c7a99c0733e6ff25ca7e465ed8aec9926cd5b05fadd5c71dc45071f897d6abe Jul 09 09:54:45.306489 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.306391 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6zbgj" event={"ID":"de11cbd5-8899-4f7f-b5db-e2236850cb4c","Type":"ContainerStarted","Data":"9a02830842579f6e88afa40adda5bd026d68f1c309ce977b8ff4a7a41a5e1a1d"} Jul 09 09:54:45.309842 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.309814 2571 generic.go:356] "Generic (PLEG): container finished" podID="1090cf01-53a3-486e-9641-90a2ca359480" containerID="2f3f677f473eebd0800ab53d7b8ce7db876319d72560afeb23250a9a2eee82f6" exitCode=0 Jul 09 09:54:45.309981 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.309856 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" event={"ID":"1090cf01-53a3-486e-9641-90a2ca359480","Type":"ContainerDied","Data":"2f3f677f473eebd0800ab53d7b8ce7db876319d72560afeb23250a9a2eee82f6"} Jul 09 09:54:45.311063 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.311038 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" event={"ID":"48bad99c-20b4-4c34-b8eb-67d0e7189c8c","Type":"ContainerStarted","Data":"7c7a99c0733e6ff25ca7e465ed8aec9926cd5b05fadd5c71dc45071f897d6abe"} Jul 09 09:54:45.326082 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.326027 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6zbgj" podStartSLOduration=1.468707086 podStartE2EDuration="6.326005473s" totalImagesPullingTime="4.857298387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:45.324373579 +0000 UTC m=+40.656857379" watchObservedRunningTime="2026-07-09 09:54:45.326005473 +0000 UTC m=+40.658489273" Jul 09 09:54:45.873318 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.873287 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw"] Jul 09 09:54:45.897518 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.897486 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw"] Jul 09 09:54:45.897683 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.897613 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:45.900685 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.900656 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 09:54:45.900808 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.900687 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 09:54:45.900929 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.900878 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hgpx6\"" Jul 09 09:54:45.900929 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.900907 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:54:45.901144 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.901112 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 09:54:45.952223 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.952181 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e920c339-7a54-4742-b438-9dddc5ca84a7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:45.952392 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.952241 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e920c339-7a54-4742-b438-9dddc5ca84a7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:45.952392 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:45.952329 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkkrs\" (UniqueName: \"kubernetes.io/projected/e920c339-7a54-4742-b438-9dddc5ca84a7-kube-api-access-hkkrs\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:46.053622 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.053584 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e920c339-7a54-4742-b438-9dddc5ca84a7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:46.053819 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.053691 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkkrs\" (UniqueName: \"kubernetes.io/projected/e920c339-7a54-4742-b438-9dddc5ca84a7-kube-api-access-hkkrs\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:46.053819 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.053806 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e920c339-7a54-4742-b438-9dddc5ca84a7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:46.054280 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.054248 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e920c339-7a54-4742-b438-9dddc5ca84a7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:46.056417 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.056392 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e920c339-7a54-4742-b438-9dddc5ca84a7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:46.065730 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.065643 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkkrs\" (UniqueName: \"kubernetes.io/projected/e920c339-7a54-4742-b438-9dddc5ca84a7-kube-api-access-hkkrs\") pod \"kube-storage-version-migrator-operator-846bf668f7-bwzqw\" (UID: \"e920c339-7a54-4742-b438-9dddc5ca84a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:46.209209 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.209175 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" Jul 09 09:54:46.231861 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.231831 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn"] Jul 09 09:54:46.261669 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.261625 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn"] Jul 09 09:54:46.261851 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.261746 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn" Jul 09 09:54:46.264459 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.264436 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vvkcj\"" Jul 09 09:54:46.318472 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.318390 2571 generic.go:356] "Generic (PLEG): container finished" podID="1090cf01-53a3-486e-9641-90a2ca359480" containerID="1bfd6deb61559b8aad421cc460b99ec2422ab6fd776dd42e84edf2ea507827a4" exitCode=0 Jul 09 09:54:46.318625 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.318466 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" event={"ID":"1090cf01-53a3-486e-9641-90a2ca359480","Type":"ContainerDied","Data":"1bfd6deb61559b8aad421cc460b99ec2422ab6fd776dd42e84edf2ea507827a4"} Jul 09 09:54:46.322854 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.322754 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498" event={"ID":"bee94409-6c0f-47f2-8107-f0e93537ce33","Type":"ContainerStarted","Data":"8ce5edd1886f4a29311b8f13d5606f10289d56b41368a24e248a150dbdc0671e"} Jul 09 09:54:46.325703 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.325678 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" event={"ID":"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4","Type":"ContainerStarted","Data":"25e7ad9e7b3bae46175a7200e8b0d4009786b0ffd90bedacdcba909795210862"} Jul 09 09:54:46.357526 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.357168 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpw4\" (UniqueName: \"kubernetes.io/projected/35ac37fb-7cad-4c81-88d0-7d796c62571c-kube-api-access-nlpw4\") pod \"network-check-source-989f8b8dc-2rmmn\" (UID: \"35ac37fb-7cad-4c81-88d0-7d796c62571c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn" Jul 09 09:54:46.377378 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.377300 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" podStartSLOduration=2.294672209 podStartE2EDuration="6.377277149s" totalImagesPullingTime="4.08260494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:46.372209517 +0000 UTC m=+41.704693316" watchObservedRunningTime="2026-07-09 09:54:46.377277149 +0000 UTC m=+41.709760948" Jul 09 09:54:46.377840 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.377816 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw"] Jul 09 09:54:46.394255 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.394203 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-fb498" podStartSLOduration=3.660610411 podStartE2EDuration="6.394183898s" totalImagesPullingTime="2.733573487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:46.394103506 +0000 UTC m=+41.726587304" watchObservedRunningTime="2026-07-09 09:54:46.394183898 +0000 UTC m=+41.726667697" Jul 09 09:54:46.458416 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.458367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpw4\" (UniqueName: \"kubernetes.io/projected/35ac37fb-7cad-4c81-88d0-7d796c62571c-kube-api-access-nlpw4\") pod \"network-check-source-989f8b8dc-2rmmn\" (UID: \"35ac37fb-7cad-4c81-88d0-7d796c62571c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn" Jul 09 09:54:46.472647 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.472608 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpw4\" (UniqueName: \"kubernetes.io/projected/35ac37fb-7cad-4c81-88d0-7d796c62571c-kube-api-access-nlpw4\") pod \"network-check-source-989f8b8dc-2rmmn\" (UID: \"35ac37fb-7cad-4c81-88d0-7d796c62571c\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn" Jul 09 09:54:46.573810 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.573700 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn" Jul 09 09:54:46.725479 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:46.725442 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn"] Jul 09 09:54:46.730362 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:46.730334 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ac37fb_7cad_4c81_88d0_7d796c62571c.slice/crio-ec56270d6b48f187c2518a9be44c5803a0d49021eba7ebc0c14b426cc7f35d38 WatchSource:0}: Error finding container ec56270d6b48f187c2518a9be44c5803a0d49021eba7ebc0c14b426cc7f35d38: Status 404 returned error can't find the container with id ec56270d6b48f187c2518a9be44c5803a0d49021eba7ebc0c14b426cc7f35d38 Jul 09 09:54:47.268118 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.268073 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:47.268603 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.268182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:47.268603 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.268260 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 09:54:47.268603 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.268389 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 09:54:47.268603 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.268400 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls podName:5b882c2a-8f11-4f25-aeb0-da5a38087884 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:55.268359736 +0000 UTC m=+50.600843529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls") pod "dns-default-wp9sj" (UID: "5b882c2a-8f11-4f25-aeb0-da5a38087884") : secret "dns-default-metrics-tls" not found Jul 09 09:54:47.268603 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.268409 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9ddf4fc-2m6lk: secret "image-registry-tls" not found Jul 09 09:54:47.268603 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.268505 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls podName:57cc59ae-e2bf-4fde-86f2-e1878ae23bb8 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:55.268487244 +0000 UTC m=+50.600971040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls") pod "image-registry-78f9ddf4fc-2m6lk" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8") : secret "image-registry-tls" not found Jul 09 09:54:47.306880 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.306635 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k2n6k"] Jul 09 09:54:47.332786 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.332756 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k2n6k"] Jul 09 09:54:47.332925 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.332809 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" event={"ID":"e920c339-7a54-4742-b438-9dddc5ca84a7","Type":"ContainerStarted","Data":"22900bdf00dc4f8ce0a5d8d29165cba8dd0951e8446ed511f1b47430735851e2"} Jul 09 09:54:47.332925 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.332840 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" event={"ID":"1090cf01-53a3-486e-9641-90a2ca359480","Type":"ContainerStarted","Data":"3d68ed2958b3a280b9b74b0e00c708c84e1bc68ad1b2aff153d54808d715c1de"} Jul 09 09:54:47.332925 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.332846 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.333275 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.333252 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn" event={"ID":"35ac37fb-7cad-4c81-88d0-7d796c62571c","Type":"ContainerStarted","Data":"ec56270d6b48f187c2518a9be44c5803a0d49021eba7ebc0c14b426cc7f35d38"} Jul 09 09:54:47.337145 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.337101 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 09:54:47.337265 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.337107 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 09:54:47.338385 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.338365 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zkpxd\"" Jul 09 09:54:47.361992 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.361926 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2mcbb" podStartSLOduration=8.187406597 podStartE2EDuration="42.361905066s" totalImagesPullingTime="34.174498469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:47.36048925 +0000 UTC m=+42.692973048" watchObservedRunningTime="2026-07-09 09:54:47.361905066 +0000 UTC m=+42.694388881" Jul 09 09:54:47.369407 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.369379 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.369620 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.369600 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.369857 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.369833 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffe1512c-1baa-4b07-996b-38dcd28ebcec-data-volume\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.369919 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.369892 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmpg\" (UniqueName: \"kubernetes.io/projected/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-api-access-dtmpg\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.369972 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.369923 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffe1512c-1baa-4b07-996b-38dcd28ebcec-crio-socket\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.471022 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.470980 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.471212 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.471086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.471212 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.471150 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 09:54:47.471212 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.471210 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:54:47.971195209 +0000 UTC m=+43.303678990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : secret "insights-runtime-extractor-tls" not found Jul 09 09:54:47.471329 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.471225 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 09:54:47.471329 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.471269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffe1512c-1baa-4b07-996b-38dcd28ebcec-data-volume\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.471329 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.471300 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:54:47.971285031 +0000 UTC m=+43.303768807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : configmap "kube-rbac-proxy" not found Jul 09 09:54:47.471430 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.471339 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmpg\" (UniqueName: \"kubernetes.io/projected/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-api-access-dtmpg\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.471430 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.471375 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffe1512c-1baa-4b07-996b-38dcd28ebcec-crio-socket\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.471568 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.471552 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffe1512c-1baa-4b07-996b-38dcd28ebcec-data-volume\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.471602 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.471573 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ffe1512c-1baa-4b07-996b-38dcd28ebcec-crio-socket\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.480214 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.480183 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmpg\" (UniqueName: \"kubernetes.io/projected/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-api-access-dtmpg\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.858090 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.858057 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m"] Jul 09 09:54:47.872555 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.872529 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:47.873104 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.872997 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m"] Jul 09 09:54:47.875563 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.875365 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 09:54:47.876003 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.875804 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 09:54:47.876003 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.875819 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 09:54:47.876003 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.875906 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rvxh2\"" Jul 09 09:54:47.876003 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.875986 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:54:47.976609 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.976523 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-config\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:47.976609 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.976580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:47.976761 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.976729 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.976854 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.976835 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:47.976908 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:47.976874 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hk8\" (UniqueName: \"kubernetes.io/projected/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-kube-api-access-j4hk8\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:47.976908 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.976893 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 09:54:47.976995 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.976965 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:54:48.976943363 +0000 UTC m=+44.309427143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : secret "insights-runtime-extractor-tls" not found Jul 09 09:54:47.976995 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.976963 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 09:54:47.977102 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:47.977002 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:54:48.976992136 +0000 UTC m=+44.309475914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : configmap "kube-rbac-proxy" not found Jul 09 09:54:48.078280 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.078006 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hk8\" (UniqueName: \"kubernetes.io/projected/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-kube-api-access-j4hk8\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:48.078280 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.078179 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-config\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:48.078280 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.078229 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:48.079179 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.079149 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-config\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:48.080958 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.080932 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:48.089409 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.089381 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hk8\" (UniqueName: \"kubernetes.io/projected/a38dd258-2b58-4b2b-ba76-86f8d4f1c474-kube-api-access-j4hk8\") pod \"service-ca-operator-5d46d9ffd4-c5t4m\" (UID: \"a38dd258-2b58-4b2b-ba76-86f8d4f1c474\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:48.184826 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.184784 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" Jul 09 09:54:48.338220 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.338112 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/0.log" Jul 09 09:54:48.338220 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.338190 2571 generic.go:356] "Generic (PLEG): container finished" podID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" containerID="f0a6e5a287b39c1883970d7191c5aa94ab0e1a397e3fafabb92e2647d94740dd" exitCode=255 Jul 09 09:54:48.338704 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.338225 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" event={"ID":"48bad99c-20b4-4c34-b8eb-67d0e7189c8c","Type":"ContainerDied","Data":"f0a6e5a287b39c1883970d7191c5aa94ab0e1a397e3fafabb92e2647d94740dd"} Jul 09 09:54:48.338704 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.338491 2571 scope.go:117] "RemoveContainer" containerID="f0a6e5a287b39c1883970d7191c5aa94ab0e1a397e3fafabb92e2647d94740dd" Jul 09 09:54:48.361111 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.361042 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m"] Jul 09 09:54:48.367989 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:54:48.367915 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38dd258_2b58_4b2b_ba76_86f8d4f1c474.slice/crio-f88851bf4c4c22a0b1ee752a8ba67957cf2fb7281f00f8a59c8f6e4cc0240feb WatchSource:0}: Error finding container f88851bf4c4c22a0b1ee752a8ba67957cf2fb7281f00f8a59c8f6e4cc0240feb: Status 404 returned error can't find the container with id f88851bf4c4c22a0b1ee752a8ba67957cf2fb7281f00f8a59c8f6e4cc0240feb Jul 09 09:54:48.506859 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.506830 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k57ns_8ff1e23f-f24b-4568-b9ee-1f0692a9afe3/dns-node-resolver/0.log" Jul 09 09:54:48.786862 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.786765 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:48.787021 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.786965 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 09:54:48.787088 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.787033 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls podName:88eeac34-fbf1-41c0-956e-7ee749407b8a nodeName:}" failed. No retries permitted until 2026-07-09 09:54:56.787012535 +0000 UTC m=+52.119496334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nsqsp" (UID: "88eeac34-fbf1-41c0-956e-7ee749407b8a") : secret "samples-operator-tls" not found Jul 09 09:54:48.887677 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.887635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:48.887843 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.887713 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:48.887843 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.887789 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:48.887843 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.887802 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 09:54:48.887943 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.887863 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:56.887846459 +0000 UTC m=+52.220330258 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : secret "router-metrics-certs-default" not found Jul 09 09:54:48.887943 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.887883 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:54:56.887873353 +0000 UTC m=+52.220357133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : configmap references non-existent config key: service-ca.crt Jul 09 09:54:48.887943 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.887893 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:48.887943 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.887927 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls podName:35d1d4bd-3fef-4335-a2b3-8085386414da nodeName:}" failed. No retries permitted until 2026-07-09 09:54:56.887918438 +0000 UTC m=+52.220402216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-q7kml" (UID: "35d1d4bd-3fef-4335-a2b3-8085386414da") : secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:48.989000 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.988962 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:48.989185 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.989099 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 09:54:48.989185 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:48.989139 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:48.989185 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.989183 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:54:50.989165501 +0000 UTC m=+46.321649278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : configmap "kube-rbac-proxy" not found Jul 09 09:54:48.989376 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.989254 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 09:54:48.989376 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:48.989326 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:54:50.989308032 +0000 UTC m=+46.321791813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : secret "insights-runtime-extractor-tls" not found Jul 09 09:54:49.344489 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:49.344447 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" event={"ID":"a38dd258-2b58-4b2b-ba76-86f8d4f1c474","Type":"ContainerStarted","Data":"f88851bf4c4c22a0b1ee752a8ba67957cf2fb7281f00f8a59c8f6e4cc0240feb"} Jul 09 09:54:49.346419 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:49.346397 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/1.log" Jul 09 09:54:49.346902 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:49.346879 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/0.log" Jul 09 09:54:49.347006 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:49.346925 2571 generic.go:356] "Generic (PLEG): container finished" podID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" containerID="c6d8169547a4d8f0e01083e7186e35ae9b694b8df00228b277f55964f9e653ee" exitCode=255 Jul 09 09:54:49.347006 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:49.346955 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" event={"ID":"48bad99c-20b4-4c34-b8eb-67d0e7189c8c","Type":"ContainerDied","Data":"c6d8169547a4d8f0e01083e7186e35ae9b694b8df00228b277f55964f9e653ee"} Jul 09 09:54:49.347081 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:49.347005 2571 scope.go:117] "RemoveContainer" containerID="f0a6e5a287b39c1883970d7191c5aa94ab0e1a397e3fafabb92e2647d94740dd" Jul 09 09:54:49.347317 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:49.347298 2571 scope.go:117] "RemoveContainer" containerID="c6d8169547a4d8f0e01083e7186e35ae9b694b8df00228b277f55964f9e653ee" Jul 09 09:54:49.347573 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:49.347531 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-xph5s_openshift-console-operator(48bad99c-20b4-4c34-b8eb-67d0e7189c8c)\"" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" podUID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" Jul 09 09:54:49.706833 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:49.706798 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l84jq_d31879ac-51f8-4f04-8591-dd2c0487c68b/node-ca/0.log" Jul 09 09:54:50.106206 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:50.106176 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6zbgj_de11cbd5-8899-4f7f-b5db-e2236850cb4c/serve-healthcheck-canary/0.log" Jul 09 09:54:50.350780 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:50.350728 2571 scope.go:117] "RemoveContainer" containerID="c6d8169547a4d8f0e01083e7186e35ae9b694b8df00228b277f55964f9e653ee" Jul 09 09:54:50.351387 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:50.350941 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-xph5s_openshift-console-operator(48bad99c-20b4-4c34-b8eb-67d0e7189c8c)\"" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" podUID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" Jul 09 09:54:51.010296 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:51.010261 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:51.010454 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:51.010373 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 09:54:51.010454 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:51.010433 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:54:55.01041835 +0000 UTC m=+50.342902130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : configmap "kube-rbac-proxy" not found Jul 09 09:54:51.010454 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:51.010378 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:51.010566 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:51.010456 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 09:54:51.010566 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:51.010509 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:54:55.010493606 +0000 UTC m=+50.342977385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : secret "insights-runtime-extractor-tls" not found Jul 09 09:54:51.355088 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:51.355060 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/1.log" Jul 09 09:54:52.193615 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.193575 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:52.193780 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.193639 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:54:52.193957 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.193945 2571 scope.go:117] "RemoveContainer" containerID="c6d8169547a4d8f0e01083e7186e35ae9b694b8df00228b277f55964f9e653ee" Jul 09 09:54:52.194108 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:52.194090 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-xph5s_openshift-console-operator(48bad99c-20b4-4c34-b8eb-67d0e7189c8c)\"" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" podUID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" Jul 09 09:54:52.360496 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.360457 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" event={"ID":"a38dd258-2b58-4b2b-ba76-86f8d4f1c474","Type":"ContainerStarted","Data":"612a75d146680f60ab3a65ddac08be78e735e1de6134b176395ac48077ce50a1"} Jul 09 09:54:52.362041 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.362015 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn" event={"ID":"35ac37fb-7cad-4c81-88d0-7d796c62571c","Type":"ContainerStarted","Data":"d63efac14677ba90619607744161848bce4b7a0da28be9d2e6c5a5c14c4f2c4e"} Jul 09 09:54:52.363495 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.363470 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" event={"ID":"e920c339-7a54-4742-b438-9dddc5ca84a7","Type":"ContainerStarted","Data":"ff798c018b929a940dc79a8bdd623bb99260f40caa08350d5ec9b63a8a3ff024"} Jul 09 09:54:52.378433 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.378386 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" podStartSLOduration=1.710343687 podStartE2EDuration="5.378368486s" totalImagesPullingTime="3.668024799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:52.377670871 +0000 UTC m=+47.710154676" watchObservedRunningTime="2026-07-09 09:54:52.378368486 +0000 UTC m=+47.710852285" Jul 09 09:54:52.397094 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.397046 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" podStartSLOduration=2.102142906 podStartE2EDuration="7.397027049s" totalImagesPullingTime="5.294884143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:52.395321051 +0000 UTC m=+47.727804851" watchObservedRunningTime="2026-07-09 09:54:52.397027049 +0000 UTC m=+47.729510848" Jul 09 09:54:52.413090 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:52.413047 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-2rmmn" podStartSLOduration=1.452078319 podStartE2EDuration="6.413032511s" totalImagesPullingTime="4.960954192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:52.411726502 +0000 UTC m=+47.744210302" watchObservedRunningTime="2026-07-09 09:54:52.413032511 +0000 UTC m=+47.745516325" Jul 09 09:54:55.052172 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:55.052106 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:55.052557 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.052237 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 09:54:55.052557 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:55.052239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:54:55.052557 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.052308 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 09:54:55.052557 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.052311 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:55:03.052293629 +0000 UTC m=+58.384777410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : configmap "kube-rbac-proxy" not found Jul 09 09:54:55.052557 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.052404 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:55:03.052387731 +0000 UTC m=+58.384871510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : secret "insights-runtime-extractor-tls" not found Jul 09 09:54:55.354563 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:55.354528 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:54:55.354724 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:55.354641 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:54:55.354724 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.354700 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 09:54:55.354801 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.354768 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls podName:5b882c2a-8f11-4f25-aeb0-da5a38087884 nodeName:}" failed. No retries permitted until 2026-07-09 09:55:11.354753114 +0000 UTC m=+66.687236894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls") pod "dns-default-wp9sj" (UID: "5b882c2a-8f11-4f25-aeb0-da5a38087884") : secret "dns-default-metrics-tls" not found Jul 09 09:54:55.354801 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.354779 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 09:54:55.354801 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.354797 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78f9ddf4fc-2m6lk: secret "image-registry-tls" not found Jul 09 09:54:55.354899 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:55.354867 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls podName:57cc59ae-e2bf-4fde-86f2-e1878ae23bb8 nodeName:}" failed. No retries permitted until 2026-07-09 09:55:11.35485381 +0000 UTC m=+66.687337587 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls") pod "image-registry-78f9ddf4fc-2m6lk" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8") : secret "image-registry-tls" not found Jul 09 09:54:56.868500 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:56.868460 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:54:56.868945 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:56.868650 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 09:54:56.868945 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:56.868746 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls podName:88eeac34-fbf1-41c0-956e-7ee749407b8a nodeName:}" failed. No retries permitted until 2026-07-09 09:55:12.868724798 +0000 UTC m=+68.201208579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-nsqsp" (UID: "88eeac34-fbf1-41c0-956e-7ee749407b8a") : secret "samples-operator-tls" not found Jul 09 09:54:56.969542 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:56.969500 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:54:56.969699 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:56.969595 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:56.969699 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:54:56.969640 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:54:56.969699 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:56.969656 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:56.969801 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:56.969723 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls podName:35d1d4bd-3fef-4335-a2b3-8085386414da nodeName:}" failed. No retries permitted until 2026-07-09 09:55:12.969707028 +0000 UTC m=+68.302190809 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-q7kml" (UID: "35d1d4bd-3fef-4335-a2b3-8085386414da") : secret "cluster-monitoring-operator-tls" not found Jul 09 09:54:56.969801 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:56.969736 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 09:54:56.969801 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:56.969779 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:55:12.969765718 +0000 UTC m=+68.302249493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : configmap references non-existent config key: service-ca.crt Jul 09 09:54:56.969801 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:54:56.969795 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs podName:ed295098-19e7-483e-88c5-20da6182ea79 nodeName:}" failed. No retries permitted until 2026-07-09 09:55:12.969788111 +0000 UTC m=+68.302271888 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs") pod "router-default-56f6dfc6c5-m7gq8" (UID: "ed295098-19e7-483e-88c5-20da6182ea79") : secret "router-metrics-certs-default" not found Jul 09 09:55:03.122104 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.122063 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:55:03.122661 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.122226 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:55:03.125241 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.123050 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ffe1512c-1baa-4b07-996b-38dcd28ebcec-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:55:03.125241 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:55:03.123210 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 09:55:03.125241 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:55:03.123275 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls podName:ffe1512c-1baa-4b07-996b-38dcd28ebcec nodeName:}" failed. No retries permitted until 2026-07-09 09:55:19.123250973 +0000 UTC m=+74.455734751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k2n6k" (UID: "ffe1512c-1baa-4b07-996b-38dcd28ebcec") : secret "insights-runtime-extractor-tls" not found Jul 09 09:55:03.154977 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.154954 2571 scope.go:117] "RemoveContainer" containerID="c6d8169547a4d8f0e01083e7186e35ae9b694b8df00228b277f55964f9e653ee" Jul 09 09:55:03.397614 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.397552 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 09:55:03.397999 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.397977 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/1.log" Jul 09 09:55:03.398115 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.398014 2571 generic.go:356] "Generic (PLEG): container finished" podID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" containerID="eba079c0b66184ed9030ae38b45bb56e6d22cee7da3138920ad21371c180d062" exitCode=255 Jul 09 09:55:03.398115 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.398055 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" event={"ID":"48bad99c-20b4-4c34-b8eb-67d0e7189c8c","Type":"ContainerDied","Data":"eba079c0b66184ed9030ae38b45bb56e6d22cee7da3138920ad21371c180d062"} Jul 09 09:55:03.398115 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.398109 2571 scope.go:117] "RemoveContainer" containerID="c6d8169547a4d8f0e01083e7186e35ae9b694b8df00228b277f55964f9e653ee" Jul 09 09:55:03.398451 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:03.398422 2571 scope.go:117] "RemoveContainer" containerID="eba079c0b66184ed9030ae38b45bb56e6d22cee7da3138920ad21371c180d062" Jul 09 09:55:03.398644 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:55:03.398622 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-xph5s_openshift-console-operator(48bad99c-20b4-4c34-b8eb-67d0e7189c8c)\"" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" podUID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" Jul 09 09:55:04.402870 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:04.402843 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 09:55:06.287746 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:06.287720 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b47sp" Jul 09 09:55:10.790342 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:10.790296 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:55:10.793324 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:10.793301 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 09:55:10.803623 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:10.803590 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49062adc-752f-4d87-aa95-db7f44411a17-metrics-certs\") pod \"network-metrics-daemon-8kpn9\" (UID: \"49062adc-752f-4d87-aa95-db7f44411a17\") " pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:55:10.891690 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:10.891654 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:55:10.894671 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:10.894652 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 09:55:10.904024 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:10.904007 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/048fd61f-ba5b-41bc-95a5-366a5734571d-original-pull-secret\") pod \"global-pull-secret-syncer-slmhl\" (UID: \"048fd61f-ba5b-41bc-95a5-366a5734571d\") " pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:55:10.992899 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:10.992866 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:55:10.995250 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:10.995231 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtrb\" (UniqueName: \"kubernetes.io/projected/2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf-kube-api-access-nhtrb\") pod \"network-check-target-w8zqv\" (UID: \"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf\") " pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:55:11.066853 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.066823 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-slmhl" Jul 09 09:55:11.078749 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.078728 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffmqr\"" Jul 09 09:55:11.082049 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.082030 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4794j\"" Jul 09 09:55:11.086091 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.086073 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8kpn9" Jul 09 09:55:11.090440 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.090420 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:55:11.213769 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.213613 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-slmhl"] Jul 09 09:55:11.216483 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:11.216455 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048fd61f_ba5b_41bc_95a5_366a5734571d.slice/crio-e0010bbe6861160e1802630b4f7a8b85b815447481a959ce26b88777d6adbae2 WatchSource:0}: Error finding container e0010bbe6861160e1802630b4f7a8b85b815447481a959ce26b88777d6adbae2: Status 404 returned error can't find the container with id e0010bbe6861160e1802630b4f7a8b85b815447481a959ce26b88777d6adbae2 Jul 09 09:55:11.396424 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.396346 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:55:11.396424 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.396399 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:55:11.398629 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.398598 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b882c2a-8f11-4f25-aeb0-da5a38087884-metrics-tls\") pod \"dns-default-wp9sj\" (UID: \"5b882c2a-8f11-4f25-aeb0-da5a38087884\") " pod="openshift-dns/dns-default-wp9sj" Jul 09 09:55:11.398734 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.398664 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"image-registry-78f9ddf4fc-2m6lk\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:55:11.422974 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.422944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-slmhl" event={"ID":"048fd61f-ba5b-41bc-95a5-366a5734571d","Type":"ContainerStarted","Data":"e0010bbe6861160e1802630b4f7a8b85b815447481a959ce26b88777d6adbae2"} Jul 09 09:55:11.441052 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.441019 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w8zqv"] Jul 09 09:55:11.442438 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.442410 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8kpn9"] Jul 09 09:55:11.443281 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:11.443253 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2473d1f3_dd3a_4aa9_9503_f5f6f67bcfcf.slice/crio-809865048ca5792588522a90e6637e321738ef05d5e2623f83d9c6547955fa84 WatchSource:0}: Error finding container 809865048ca5792588522a90e6637e321738ef05d5e2623f83d9c6547955fa84: Status 404 returned error can't find the container with id 809865048ca5792588522a90e6637e321738ef05d5e2623f83d9c6547955fa84 Jul 09 09:55:11.445755 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:11.445736 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49062adc_752f_4d87_aa95_db7f44411a17.slice/crio-25e80c235eb39f25de87e2ef9c719e18675c6e67693b6f605420a93783e6f6a1 WatchSource:0}: Error finding container 25e80c235eb39f25de87e2ef9c719e18675c6e67693b6f605420a93783e6f6a1: Status 404 returned error can't find the container with id 25e80c235eb39f25de87e2ef9c719e18675c6e67693b6f605420a93783e6f6a1 Jul 09 09:55:11.577855 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.577825 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jpdnh\"" Jul 09 09:55:11.586047 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.586027 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:55:11.594642 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.594624 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n6pb2\"" Jul 09 09:55:11.602824 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.602801 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wp9sj" Jul 09 09:55:11.722629 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.722591 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78f9ddf4fc-2m6lk"] Jul 09 09:55:11.726298 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:11.726274 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57cc59ae_e2bf_4fde_86f2_e1878ae23bb8.slice/crio-6c31808b7fa654dad5d562f1828336e7479a666556c4010c48d24fcea8517968 WatchSource:0}: Error finding container 6c31808b7fa654dad5d562f1828336e7479a666556c4010c48d24fcea8517968: Status 404 returned error can't find the container with id 6c31808b7fa654dad5d562f1828336e7479a666556c4010c48d24fcea8517968 Jul 09 09:55:11.740016 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:11.739993 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wp9sj"] Jul 09 09:55:11.742819 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:11.742796 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b882c2a_8f11_4f25_aeb0_da5a38087884.slice/crio-bed55c615ff8479db4925294725d0c12c19723479ea65f9eabd96e24b2d33ebb WatchSource:0}: Error finding container bed55c615ff8479db4925294725d0c12c19723479ea65f9eabd96e24b2d33ebb: Status 404 returned error can't find the container with id bed55c615ff8479db4925294725d0c12c19723479ea65f9eabd96e24b2d33ebb Jul 09 09:55:12.193434 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.193393 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:55:12.193904 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.193457 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:55:12.193904 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.193889 2571 scope.go:117] "RemoveContainer" containerID="eba079c0b66184ed9030ae38b45bb56e6d22cee7da3138920ad21371c180d062" Jul 09 09:55:12.194142 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:55:12.194094 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-xph5s_openshift-console-operator(48bad99c-20b4-4c34-b8eb-67d0e7189c8c)\"" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" podUID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" Jul 09 09:55:12.429817 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.429776 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" event={"ID":"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8","Type":"ContainerStarted","Data":"5ab7b8f1a24bfaa319d2063e97938ddf8fabe24a87f4511296dd6b87d9d96d14"} Jul 09 09:55:12.429997 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.429841 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" event={"ID":"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8","Type":"ContainerStarted","Data":"6c31808b7fa654dad5d562f1828336e7479a666556c4010c48d24fcea8517968"} Jul 09 09:55:12.429997 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.429982 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:55:12.431492 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.431461 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8kpn9" event={"ID":"49062adc-752f-4d87-aa95-db7f44411a17","Type":"ContainerStarted","Data":"25e80c235eb39f25de87e2ef9c719e18675c6e67693b6f605420a93783e6f6a1"} Jul 09 09:55:12.432861 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.432838 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wp9sj" event={"ID":"5b882c2a-8f11-4f25-aeb0-da5a38087884","Type":"ContainerStarted","Data":"bed55c615ff8479db4925294725d0c12c19723479ea65f9eabd96e24b2d33ebb"} Jul 09 09:55:12.434622 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.434598 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w8zqv" event={"ID":"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf","Type":"ContainerStarted","Data":"d09cdd0e4bcbcdbb994cce7558d1e168e3369928c9b9e88cb26947e2fa020cc4"} Jul 09 09:55:12.434735 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.434637 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w8zqv" event={"ID":"2473d1f3-dd3a-4aa9-9503-f5f6f67bcfcf","Type":"ContainerStarted","Data":"809865048ca5792588522a90e6637e321738ef05d5e2623f83d9c6547955fa84"} Jul 09 09:55:12.434803 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.434748 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:55:12.452769 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.452531 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" podStartSLOduration=66.452512143 podStartE2EDuration="1m6.452512143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:12.451950608 +0000 UTC m=+67.784434407" watchObservedRunningTime="2026-07-09 09:55:12.452512143 +0000 UTC m=+67.784995924" Jul 09 09:55:12.469887 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.469827 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w8zqv" podStartSLOduration=67.46980864 podStartE2EDuration="1m7.46980864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:12.469779232 +0000 UTC m=+67.802263024" watchObservedRunningTime="2026-07-09 09:55:12.46980864 +0000 UTC m=+67.802292440" Jul 09 09:55:12.910595 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.910561 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:55:12.913446 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:12.913418 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/88eeac34-fbf1-41c0-956e-7ee749407b8a-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-nsqsp\" (UID: \"88eeac34-fbf1-41c0-956e-7ee749407b8a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:55:13.011275 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.011193 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:55:13.011416 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.011278 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:13.011416 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.011329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:13.012008 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.011976 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed295098-19e7-483e-88c5-20da6182ea79-service-ca-bundle\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:13.013767 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.013741 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/35d1d4bd-3fef-4335-a2b3-8085386414da-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-q7kml\" (UID: \"35d1d4bd-3fef-4335-a2b3-8085386414da\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:55:13.013767 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.013759 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed295098-19e7-483e-88c5-20da6182ea79-metrics-certs\") pod \"router-default-56f6dfc6c5-m7gq8\" (UID: \"ed295098-19e7-483e-88c5-20da6182ea79\") " pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:13.105786 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.105757 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xjjkj\"" Jul 09 09:55:13.113259 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.113240 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" Jul 09 09:55:13.153944 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.153734 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dstrq\"" Jul 09 09:55:13.161552 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.161531 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" Jul 09 09:55:13.167100 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.166894 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tqxqm\"" Jul 09 09:55:13.175406 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.175379 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:13.275493 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.275230 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp"] Jul 09 09:55:13.335504 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.335288 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml"] Jul 09 09:55:13.338313 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:13.338280 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d1d4bd_3fef_4335_a2b3_8085386414da.slice/crio-5839078a5d2ffb22e324ca802017de655859e0448f8629b3cf92fc4f6ad36616 WatchSource:0}: Error finding container 5839078a5d2ffb22e324ca802017de655859e0448f8629b3cf92fc4f6ad36616: Status 404 returned error can't find the container with id 5839078a5d2ffb22e324ca802017de655859e0448f8629b3cf92fc4f6ad36616 Jul 09 09:55:13.362238 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.362061 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56f6dfc6c5-m7gq8"] Jul 09 09:55:13.364708 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:13.364679 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded295098_19e7_483e_88c5_20da6182ea79.slice/crio-213cdbc90acdda0a51f20ae5cc386c041022b3a1c54dd654c70112a86b1c36dd WatchSource:0}: Error finding container 213cdbc90acdda0a51f20ae5cc386c041022b3a1c54dd654c70112a86b1c36dd: Status 404 returned error can't find the container with id 213cdbc90acdda0a51f20ae5cc386c041022b3a1c54dd654c70112a86b1c36dd Jul 09 09:55:13.441676 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.441636 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8kpn9" event={"ID":"49062adc-752f-4d87-aa95-db7f44411a17","Type":"ContainerStarted","Data":"1c9e4b972de40c4deb545bb26c446d6629c81d373cac604b63e7bf6378a4cf35"} Jul 09 09:55:13.441790 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.441711 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8kpn9" event={"ID":"49062adc-752f-4d87-aa95-db7f44411a17","Type":"ContainerStarted","Data":"25bf34981c4124393addc8383f9e4d0c12b170f86a7858ca22f34acc979a2112"} Jul 09 09:55:13.445329 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.444801 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" event={"ID":"ed295098-19e7-483e-88c5-20da6182ea79","Type":"ContainerStarted","Data":"d2ee298c4ed30ba098f372014c666721ed41b74c2f469dad8c8e8a15b262534b"} Jul 09 09:55:13.445329 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.444851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" event={"ID":"ed295098-19e7-483e-88c5-20da6182ea79","Type":"ContainerStarted","Data":"213cdbc90acdda0a51f20ae5cc386c041022b3a1c54dd654c70112a86b1c36dd"} Jul 09 09:55:13.447210 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.447118 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" event={"ID":"35d1d4bd-3fef-4335-a2b3-8085386414da","Type":"ContainerStarted","Data":"5839078a5d2ffb22e324ca802017de655859e0448f8629b3cf92fc4f6ad36616"} Jul 09 09:55:13.449013 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.448931 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" event={"ID":"88eeac34-fbf1-41c0-956e-7ee749407b8a","Type":"ContainerStarted","Data":"74caacc0e4d11cf445921c86873c9a77ede0638374428cd7aafb3320f69d4f46"} Jul 09 09:55:13.458656 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.458591 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8kpn9" podStartSLOduration=67.389560635 podStartE2EDuration="1m8.458571744s" totalImagesPullingTime="1.069011109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:13.458377807 +0000 UTC m=+68.790861606" watchObservedRunningTime="2026-07-09 09:55:13.458571744 +0000 UTC m=+68.791055543" Jul 09 09:55:13.478851 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:13.478718 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" podStartSLOduration=33.478703555 podStartE2EDuration="33.478703555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:13.477067538 +0000 UTC m=+68.809551350" watchObservedRunningTime="2026-07-09 09:55:13.478703555 +0000 UTC m=+68.811187353" Jul 09 09:55:14.176211 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:14.176153 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:14.179412 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:14.179182 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:14.453776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:14.453692 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:14.455370 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:14.455348 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56f6dfc6c5-m7gq8" Jul 09 09:55:15.816262 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.816227 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh"] Jul 09 09:55:15.819844 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.819824 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.825276 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.825253 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 09:55:15.825387 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.825254 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 09:55:15.825602 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.825582 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 09:55:15.826676 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.826626 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 09 09:55:15.838041 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.838010 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh"] Jul 09 09:55:15.844588 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.844559 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/96b4465a-a119-4005-bab1-4b59f9007ab5-klusterlet-config\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.844749 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.844723 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96b4465a-a119-4005-bab1-4b59f9007ab5-tmp\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.844880 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.844767 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbkxb\" (UniqueName: \"kubernetes.io/projected/96b4465a-a119-4005-bab1-4b59f9007ab5-kube-api-access-jbkxb\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.945469 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.945440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96b4465a-a119-4005-bab1-4b59f9007ab5-tmp\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.945638 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.945610 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkxb\" (UniqueName: \"kubernetes.io/projected/96b4465a-a119-4005-bab1-4b59f9007ab5-kube-api-access-jbkxb\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.945719 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.945689 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/96b4465a-a119-4005-bab1-4b59f9007ab5-klusterlet-config\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.945891 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.945870 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96b4465a-a119-4005-bab1-4b59f9007ab5-tmp\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.948492 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.948473 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/96b4465a-a119-4005-bab1-4b59f9007ab5-klusterlet-config\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:15.966716 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:15.966693 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkxb\" (UniqueName: \"kubernetes.io/projected/96b4465a-a119-4005-bab1-4b59f9007ab5-kube-api-access-jbkxb\") pod \"klusterlet-addon-workmgr-cf5598d7b-lcfhh\" (UID: \"96b4465a-a119-4005-bab1-4b59f9007ab5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:16.130764 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:16.130696 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:18.084231 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.083799 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh"] Jul 09 09:55:18.086290 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:18.086257 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b4465a_a119_4005_bab1_4b59f9007ab5.slice/crio-87b4b7537f8bce2220c49274d2454982b4a63442755231e3bbdec82af6ebe141 WatchSource:0}: Error finding container 87b4b7537f8bce2220c49274d2454982b4a63442755231e3bbdec82af6ebe141: Status 404 returned error can't find the container with id 87b4b7537f8bce2220c49274d2454982b4a63442755231e3bbdec82af6ebe141 Jul 09 09:55:18.470023 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.469924 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wp9sj" event={"ID":"5b882c2a-8f11-4f25-aeb0-da5a38087884","Type":"ContainerStarted","Data":"6b81cd7a53f2353634d6c1970a6ad3db1f6c708331514822a2eb61923cbd9274"} Jul 09 09:55:18.470023 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.469992 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wp9sj" event={"ID":"5b882c2a-8f11-4f25-aeb0-da5a38087884","Type":"ContainerStarted","Data":"87c24abd8ff38bd4cbb4c40428de1ed26bdfeddba57bec032e227ef7bf146724"} Jul 09 09:55:18.470263 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.470067 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wp9sj" Jul 09 09:55:18.471903 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.471876 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" event={"ID":"35d1d4bd-3fef-4335-a2b3-8085386414da","Type":"ContainerStarted","Data":"80dba2a31ad6d1a9d4fd583f3408d9febc1bc671df7e2cff3e097b99bb4f84ef"} Jul 09 09:55:18.473843 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.473815 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" event={"ID":"88eeac34-fbf1-41c0-956e-7ee749407b8a","Type":"ContainerStarted","Data":"8260d152d7f93030762c80225d2fbcef01a66b2458156b63120d4227edd01857"} Jul 09 09:55:18.473934 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.473862 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" event={"ID":"88eeac34-fbf1-41c0-956e-7ee749407b8a","Type":"ContainerStarted","Data":"cb39592e56da601daf1fd67b9306c301eb07b185730795dcd208db93747c6b0b"} Jul 09 09:55:18.475250 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.475229 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-slmhl" event={"ID":"048fd61f-ba5b-41bc-95a5-366a5734571d","Type":"ContainerStarted","Data":"2a6b22a18be9a547f98d1a071a5b4e8a67d18112a047d892ae3d7f996e1d8337"} Jul 09 09:55:18.476360 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.476337 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" event={"ID":"96b4465a-a119-4005-bab1-4b59f9007ab5","Type":"ContainerStarted","Data":"87b4b7537f8bce2220c49274d2454982b4a63442755231e3bbdec82af6ebe141"} Jul 09 09:55:18.490577 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.490537 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wp9sj" podStartSLOduration=33.302114493 podStartE2EDuration="39.490523917s" totalImagesPullingTime="6.188409424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:18.489191946 +0000 UTC m=+73.821675757" watchObservedRunningTime="2026-07-09 09:55:18.490523917 +0000 UTC m=+73.823007714" Jul 09 09:55:18.509658 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.509598 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-slmhl" podStartSLOduration=66.794945858 podStartE2EDuration="1m13.509580642s" totalImagesPullingTime="6.714634784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:18.509309113 +0000 UTC m=+73.841792910" watchObservedRunningTime="2026-07-09 09:55:18.509580642 +0000 UTC m=+73.842064441" Jul 09 09:55:18.531538 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.531412 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-nsqsp" podStartSLOduration=33.936592089 podStartE2EDuration="38.531392908s" totalImagesPullingTime="4.594800819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:18.531320481 +0000 UTC m=+73.863804281" watchObservedRunningTime="2026-07-09 09:55:18.531392908 +0000 UTC m=+73.863876707" Jul 09 09:55:18.551472 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:18.551421 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-q7kml" podStartSLOduration=33.96027763 podStartE2EDuration="38.551405822s" totalImagesPullingTime="4.591128192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:18.55067732 +0000 UTC m=+73.883161119" watchObservedRunningTime="2026-07-09 09:55:18.551405822 +0000 UTC m=+73.883889619" Jul 09 09:55:19.171750 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:19.171710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:55:19.174686 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:19.174636 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ffe1512c-1baa-4b07-996b-38dcd28ebcec-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k2n6k\" (UID: \"ffe1512c-1baa-4b07-996b-38dcd28ebcec\") " pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:55:19.446159 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:19.446062 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zkpxd\"" Jul 09 09:55:19.453540 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:19.453513 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k2n6k" Jul 09 09:55:19.638424 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:19.638398 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k2n6k"] Jul 09 09:55:19.641074 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:19.641045 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe1512c_1baa_4b07_996b_38dcd28ebcec.slice/crio-d2de057aac0b9b403189382ae403e6a4f0407dc7c71cee365df71e888d897032 WatchSource:0}: Error finding container d2de057aac0b9b403189382ae403e6a4f0407dc7c71cee365df71e888d897032: Status 404 returned error can't find the container with id d2de057aac0b9b403189382ae403e6a4f0407dc7c71cee365df71e888d897032 Jul 09 09:55:20.488906 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:20.488870 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2n6k" event={"ID":"ffe1512c-1baa-4b07-996b-38dcd28ebcec","Type":"ContainerStarted","Data":"d2de057aac0b9b403189382ae403e6a4f0407dc7c71cee365df71e888d897032"} Jul 09 09:55:22.499709 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:22.499644 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" event={"ID":"96b4465a-a119-4005-bab1-4b59f9007ab5","Type":"ContainerStarted","Data":"d21b947bb911468dc4728d12af3fb140b3a8275c51480a1bbddc631c1ef6979a"} Jul 09 09:55:22.500197 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:22.500167 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:22.501739 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:22.501717 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" Jul 09 09:55:22.519883 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:22.519823 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-cf5598d7b-lcfhh" podStartSLOduration=3.313394169 podStartE2EDuration="7.519802777s" totalImagesPullingTime="4.206408608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:22.519159516 +0000 UTC m=+77.851643313" watchObservedRunningTime="2026-07-09 09:55:22.519802777 +0000 UTC m=+77.852286577" Jul 09 09:55:23.154863 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:23.154830 2571 scope.go:117] "RemoveContainer" containerID="eba079c0b66184ed9030ae38b45bb56e6d22cee7da3138920ad21371c180d062" Jul 09 09:55:23.155059 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:55:23.155036 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-xph5s_openshift-console-operator(48bad99c-20b4-4c34-b8eb-67d0e7189c8c)\"" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" podUID="48bad99c-20b4-4c34-b8eb-67d0e7189c8c" Jul 09 09:55:26.516387 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:26.516353 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2n6k" event={"ID":"ffe1512c-1baa-4b07-996b-38dcd28ebcec","Type":"ContainerStarted","Data":"b6d31e2dd96f26aba41523c3736a831388da29da6fe63673bc3365bf73071787"} Jul 09 09:55:27.522271 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:27.522159 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2n6k" event={"ID":"ffe1512c-1baa-4b07-996b-38dcd28ebcec","Type":"ContainerStarted","Data":"e246421a51e09dc61458dbc49d5b6f71f9c0ecd8cd2d3265571c9a85f6d29d9a"} Jul 09 09:55:28.486215 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:28.486187 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wp9sj" Jul 09 09:55:29.532631 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:29.532596 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k2n6k" event={"ID":"ffe1512c-1baa-4b07-996b-38dcd28ebcec","Type":"ContainerStarted","Data":"4f5a507471c379a1502526be5225b45c1021842fc826dd840f1f5d1191e8901d"} Jul 09 09:55:29.552920 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:29.552877 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k2n6k" podStartSLOduration=32.991156888 podStartE2EDuration="42.552862568s" totalImagesPullingTime="9.56170568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:54:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:29.551781074 +0000 UTC m=+84.884264872" watchObservedRunningTime="2026-07-09 09:55:29.552862568 +0000 UTC m=+84.885346365" Jul 09 09:55:32.050185 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.050151 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w4dq8"] Jul 09 09:55:32.053242 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.053220 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.055792 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.055764 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ffn49\"" Jul 09 09:55:32.055948 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.055930 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 09:55:32.056003 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.055936 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 09:55:32.056809 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.056790 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 09:55:32.056900 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.056796 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 09:55:32.075424 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075393 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-root\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.075424 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075430 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.075630 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075457 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-wtmp\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.075630 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075495 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-tls\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.075630 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075519 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-textfile\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.075630 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075539 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-sys\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.075630 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5781a5e1-10e2-421d-9a75-9be4dfa352e5-metrics-client-ca\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.075630 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075582 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.075888 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.075677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4gqs\" (UniqueName: \"kubernetes.io/projected/5781a5e1-10e2-421d-9a75-9be4dfa352e5-kube-api-access-d4gqs\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176017 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.175986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-wtmp\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176017 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-tls\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176300 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176055 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-textfile\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176300 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176079 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-sys\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176300 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176108 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5781a5e1-10e2-421d-9a75-9be4dfa352e5-metrics-client-ca\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176300 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176189 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-wtmp\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176300 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:55:32.176194 2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 09:55:32.176300 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176215 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-sys\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176300 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176237 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176300 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:55:32.176262 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-tls podName:5781a5e1-10e2-421d-9a75-9be4dfa352e5 nodeName:}" failed. No retries permitted until 2026-07-09 09:55:32.676241656 +0000 UTC m=+88.008725438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-tls") pod "node-exporter-w4dq8" (UID: "5781a5e1-10e2-421d-9a75-9be4dfa352e5") : secret "node-exporter-tls" not found Jul 09 09:55:32.176639 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4gqs\" (UniqueName: \"kubernetes.io/projected/5781a5e1-10e2-421d-9a75-9be4dfa352e5-kube-api-access-d4gqs\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176639 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-root\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176639 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176426 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5781a5e1-10e2-421d-9a75-9be4dfa352e5-root\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176639 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176467 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-textfile\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176639 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176499 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176785 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176713 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5781a5e1-10e2-421d-9a75-9be4dfa352e5-metrics-client-ca\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.176818 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.176792 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.178771 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.178747 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.185737 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.185717 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4gqs\" (UniqueName: \"kubernetes.io/projected/5781a5e1-10e2-421d-9a75-9be4dfa352e5-kube-api-access-d4gqs\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.681170 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.681110 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-tls\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.683417 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.683393 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5781a5e1-10e2-421d-9a75-9be4dfa352e5-node-exporter-tls\") pod \"node-exporter-w4dq8\" (UID: \"5781a5e1-10e2-421d-9a75-9be4dfa352e5\") " pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.962714 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:32.962632 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w4dq8" Jul 09 09:55:32.973990 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:32.973959 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5781a5e1_10e2_421d_9a75_9be4dfa352e5.slice/crio-e8030b4a18ebea0a701a67ab74fc6471c31a29409a042d979e069f8ec6f45aaa WatchSource:0}: Error finding container e8030b4a18ebea0a701a67ab74fc6471c31a29409a042d979e069f8ec6f45aaa: Status 404 returned error can't find the container with id e8030b4a18ebea0a701a67ab74fc6471c31a29409a042d979e069f8ec6f45aaa Jul 09 09:55:33.454417 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:33.454391 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:55:33.549336 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:33.549304 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4dq8" event={"ID":"5781a5e1-10e2-421d-9a75-9be4dfa352e5","Type":"ContainerStarted","Data":"e8030b4a18ebea0a701a67ab74fc6471c31a29409a042d979e069f8ec6f45aaa"} Jul 09 09:55:34.554864 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:34.554826 2571 generic.go:356] "Generic (PLEG): container finished" podID="5781a5e1-10e2-421d-9a75-9be4dfa352e5" containerID="1763ab1ab81abff8069f97eccd0a544d84a7115fc703a96689bfb7574dac669b" exitCode=0 Jul 09 09:55:34.555260 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:34.554918 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4dq8" event={"ID":"5781a5e1-10e2-421d-9a75-9be4dfa352e5","Type":"ContainerDied","Data":"1763ab1ab81abff8069f97eccd0a544d84a7115fc703a96689bfb7574dac669b"} Jul 09 09:55:35.560958 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:35.560920 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4dq8" event={"ID":"5781a5e1-10e2-421d-9a75-9be4dfa352e5","Type":"ContainerStarted","Data":"736b4fc450f30846dfdf45ce1721f3873c377d5474e345d47f6f9970f07da42d"} Jul 09 09:55:35.561370 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:35.560985 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4dq8" event={"ID":"5781a5e1-10e2-421d-9a75-9be4dfa352e5","Type":"ContainerStarted","Data":"bbbadadfa42b9231beb186c4deb74b93e14a433d96bd9176f6abda7d00b362cd"} Jul 09 09:55:35.584569 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:35.584520 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w4dq8" podStartSLOduration=2.513266956 podStartE2EDuration="3.584504592s" totalImagesPullingTime="1.007064064s" totalInitContainerRuntime="64.173572ms" isStatefulPod=true podCreationTimestamp="2026-07-09 09:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:35.582754225 +0000 UTC m=+90.915238022" watchObservedRunningTime="2026-07-09 09:55:35.584504592 +0000 UTC m=+90.916988412" Jul 09 09:55:36.154683 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:36.154651 2571 scope.go:117] "RemoveContainer" containerID="eba079c0b66184ed9030ae38b45bb56e6d22cee7da3138920ad21371c180d062" Jul 09 09:55:36.566876 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:36.566848 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 09:55:36.567332 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:36.566976 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" event={"ID":"48bad99c-20b4-4c34-b8eb-67d0e7189c8c","Type":"ContainerStarted","Data":"b707d25bcb0c084d270a35beba9ebe46dd3ab88e34c7204920686b7ae7374d8d"} Jul 09 09:55:36.567452 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:36.567417 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:55:36.586916 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:36.586870 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" podStartSLOduration=52.791357188 podStartE2EDuration="55.586854951s" totalImagesPullingTime="2.795497763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:36.585784969 +0000 UTC m=+91.918268768" watchObservedRunningTime="2026-07-09 09:55:36.586854951 +0000 UTC m=+91.919338748" Jul 09 09:55:36.679102 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:36.679072 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-xph5s" Jul 09 09:55:39.512582 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:39.512544 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78f9ddf4fc-2m6lk"] Jul 09 09:55:43.452182 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:43.452149 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w8zqv" Jul 09 09:55:47.521007 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.520972 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f84655c86-v44zh"] Jul 09 09:55:47.541858 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.541830 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f84655c86-v44zh"] Jul 09 09:55:47.542004 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.541927 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.544861 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.544837 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cjkjq\"" Jul 09 09:55:47.545002 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.544870 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 09:55:47.545002 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.544877 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 09:55:47.545002 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.544872 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 09:55:47.546020 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.545944 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 09:55:47.546172 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.546009 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 09:55:47.546172 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.546030 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 09:55:47.546172 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.546047 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 09:55:47.605975 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.605938 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jtx\" (UniqueName: \"kubernetes.io/projected/82fc126b-f026-4812-80ac-c604395fde81-kube-api-access-s4jtx\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.606162 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.605985 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-oauth-config\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.606162 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.606072 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-service-ca\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.606162 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.606115 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-console-config\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.606290 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.606182 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-serving-cert\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.606290 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.606209 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-oauth-serving-cert\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.706722 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.706671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jtx\" (UniqueName: \"kubernetes.io/projected/82fc126b-f026-4812-80ac-c604395fde81-kube-api-access-s4jtx\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.706722 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.706730 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-oauth-config\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.706958 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.706777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-service-ca\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.706958 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.706807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-console-config\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.706958 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.706834 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-serving-cert\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.706958 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.706858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-oauth-serving-cert\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.707708 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.707677 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-service-ca\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.707708 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.707694 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-console-config\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.707847 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.707725 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-oauth-serving-cert\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.709348 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.709329 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-serving-cert\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.709426 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.709364 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-oauth-config\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.716541 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.716521 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jtx\" (UniqueName: \"kubernetes.io/projected/82fc126b-f026-4812-80ac-c604395fde81-kube-api-access-s4jtx\") pod \"console-f84655c86-v44zh\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:47.852061 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:47.852029 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:48.005843 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:48.005810 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f84655c86-v44zh"] Jul 09 09:55:48.009061 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:48.009035 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fc126b_f026_4812_80ac_c604395fde81.slice/crio-ceff84efd61be1960f28fd1a815b3a76622502fbdfb68edd3cfabf1a13db14eb WatchSource:0}: Error finding container ceff84efd61be1960f28fd1a815b3a76622502fbdfb68edd3cfabf1a13db14eb: Status 404 returned error can't find the container with id ceff84efd61be1960f28fd1a815b3a76622502fbdfb68edd3cfabf1a13db14eb Jul 09 09:55:48.612937 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:48.612902 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f84655c86-v44zh" event={"ID":"82fc126b-f026-4812-80ac-c604395fde81","Type":"ContainerStarted","Data":"ceff84efd61be1960f28fd1a815b3a76622502fbdfb68edd3cfabf1a13db14eb"} Jul 09 09:55:51.389015 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.388982 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86ddb48d55-tqmht"] Jul 09 09:55:51.413337 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.413308 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86ddb48d55-tqmht"] Jul 09 09:55:51.413482 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.413411 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.422056 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.422030 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 09:55:51.543627 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.543594 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-config\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.543778 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.543644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-oauth-config\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.543778 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.543688 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-service-ca\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.543778 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.543708 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-oauth-serving-cert\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.543778 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.543726 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-trusted-ca-bundle\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.543778 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.543746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-serving-cert\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.543976 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.543801 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djqpx\" (UniqueName: \"kubernetes.io/projected/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-kube-api-access-djqpx\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.644664 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.644605 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-config\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.644777 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.644688 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-oauth-config\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.644777 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.644736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-service-ca\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.644777 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.644761 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-oauth-serving-cert\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.644935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.644788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-trusted-ca-bundle\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.644935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.644821 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-serving-cert\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.644935 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.644859 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqpx\" (UniqueName: \"kubernetes.io/projected/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-kube-api-access-djqpx\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.645539 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.645449 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-config\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.645670 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.645642 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-service-ca\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.646067 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.646039 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-trusted-ca-bundle\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.646238 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.646221 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-oauth-serving-cert\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.647443 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.647420 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-oauth-config\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.647920 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.647903 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-serving-cert\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.654295 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.654277 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqpx\" (UniqueName: \"kubernetes.io/projected/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-kube-api-access-djqpx\") pod \"console-86ddb48d55-tqmht\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.724386 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.724175 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:55:51.878865 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:51.878841 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86ddb48d55-tqmht"] Jul 09 09:55:51.881221 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:55:51.881188 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a28ac20_26d3_4922_8422_f3fe5cc3e8cb.slice/crio-82eab90962545433196826b60205a0272a4cd32e4bf10cb9324cfcc2fc2312d4 WatchSource:0}: Error finding container 82eab90962545433196826b60205a0272a4cd32e4bf10cb9324cfcc2fc2312d4: Status 404 returned error can't find the container with id 82eab90962545433196826b60205a0272a4cd32e4bf10cb9324cfcc2fc2312d4 Jul 09 09:55:52.630317 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:52.630275 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86ddb48d55-tqmht" event={"ID":"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb","Type":"ContainerStarted","Data":"ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f"} Jul 09 09:55:52.630770 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:52.630371 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86ddb48d55-tqmht" event={"ID":"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb","Type":"ContainerStarted","Data":"82eab90962545433196826b60205a0272a4cd32e4bf10cb9324cfcc2fc2312d4"} Jul 09 09:55:52.631712 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:52.631687 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f84655c86-v44zh" event={"ID":"82fc126b-f026-4812-80ac-c604395fde81","Type":"ContainerStarted","Data":"666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669"} Jul 09 09:55:52.651025 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:52.650975 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86ddb48d55-tqmht" podStartSLOduration=1.650959473 podStartE2EDuration="1.650959473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:52.650317815 +0000 UTC m=+107.982801624" watchObservedRunningTime="2026-07-09 09:55:52.650959473 +0000 UTC m=+107.983443271" Jul 09 09:55:52.672749 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:52.672698 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f84655c86-v44zh" podStartSLOduration=2.107502097 podStartE2EDuration="5.672683553s" totalImagesPullingTime="3.565181456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:52.671370602 +0000 UTC m=+108.003854401" watchObservedRunningTime="2026-07-09 09:55:52.672683553 +0000 UTC m=+108.005167351" Jul 09 09:55:57.853299 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:57.853264 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:57.853748 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:57.853326 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:57.857909 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:57.857888 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:55:58.658023 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:55:58.657997 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:56:01.725083 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:01.725049 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:56:01.725542 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:01.725107 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:56:01.730039 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:01.730014 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:56:02.672239 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:02.672213 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:56:02.724548 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:02.724518 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f84655c86-v44zh"] Jul 09 09:56:04.535718 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.535661 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" podUID="57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" containerName="registry" containerID="cri-o://5ab7b8f1a24bfaa319d2063e97938ddf8fabe24a87f4511296dd6b87d9d96d14" gracePeriod=30 Jul 09 09:56:04.677304 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.677272 2571 generic.go:356] "Generic (PLEG): container finished" podID="57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" containerID="5ab7b8f1a24bfaa319d2063e97938ddf8fabe24a87f4511296dd6b87d9d96d14" exitCode=0 Jul 09 09:56:04.677449 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.677338 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" event={"ID":"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8","Type":"ContainerDied","Data":"5ab7b8f1a24bfaa319d2063e97938ddf8fabe24a87f4511296dd6b87d9d96d14"} Jul 09 09:56:04.777607 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.777582 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:56:04.965117 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965083 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-ca-trust-extracted\") pod \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " Jul 09 09:56:04.965117 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965119 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-certificates\") pod \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " Jul 09 09:56:04.965347 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965158 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-bound-sa-token\") pod \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " Jul 09 09:56:04.965388 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965337 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlqpc\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-kube-api-access-rlqpc\") pod \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " Jul 09 09:56:04.965388 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965368 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") pod \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " Jul 09 09:56:04.965462 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965409 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-trusted-ca\") pod \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " Jul 09 09:56:04.965509 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965491 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-installation-pull-secrets\") pod \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " Jul 09 09:56:04.965573 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965513 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:56:04.965573 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965524 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-image-registry-private-configuration\") pod \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\" (UID: \"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8\") " Jul 09 09:56:04.965939 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.965908 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:56:04.966317 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.966295 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-trusted-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:04.966434 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.966322 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-certificates\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:04.967837 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.967814 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-kube-api-access-rlqpc" (OuterVolumeSpecName: "kube-api-access-rlqpc") pod "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8"). InnerVolumeSpecName "kube-api-access-rlqpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:56:04.967973 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.967924 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:56:04.968039 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.968011 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:56:04.968086 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.968030 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:56:04.968195 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.968170 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:56:04.974926 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:04.974903 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" (UID: "57cc59ae-e2bf-4fde-86f2-e1878ae23bb8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:56:05.067455 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.067427 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-installation-pull-secrets\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:05.067455 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.067450 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-image-registry-private-configuration\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:05.067455 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.067460 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-ca-trust-extracted\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:05.067672 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.067468 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-bound-sa-token\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:05.067672 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.067477 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqpc\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-kube-api-access-rlqpc\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:05.067672 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.067487 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8-registry-tls\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:05.682505 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.682465 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" event={"ID":"57cc59ae-e2bf-4fde-86f2-e1878ae23bb8","Type":"ContainerDied","Data":"6c31808b7fa654dad5d562f1828336e7479a666556c4010c48d24fcea8517968"} Jul 09 09:56:05.682505 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.682499 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78f9ddf4fc-2m6lk" Jul 09 09:56:05.683051 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.682555 2571 scope.go:117] "RemoveContainer" containerID="5ab7b8f1a24bfaa319d2063e97938ddf8fabe24a87f4511296dd6b87d9d96d14" Jul 09 09:56:05.709155 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.709105 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78f9ddf4fc-2m6lk"] Jul 09 09:56:05.714892 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:05.714867 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78f9ddf4fc-2m6lk"] Jul 09 09:56:07.159529 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:07.159499 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" path="/var/lib/kubelet/pods/57cc59ae-e2bf-4fde-86f2-e1878ae23bb8/volumes" Jul 09 09:56:07.692394 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:07.692358 2571 generic.go:356] "Generic (PLEG): container finished" podID="e920c339-7a54-4742-b438-9dddc5ca84a7" containerID="ff798c018b929a940dc79a8bdd623bb99260f40caa08350d5ec9b63a8a3ff024" exitCode=0 Jul 09 09:56:07.692582 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:07.692397 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" event={"ID":"e920c339-7a54-4742-b438-9dddc5ca84a7","Type":"ContainerDied","Data":"ff798c018b929a940dc79a8bdd623bb99260f40caa08350d5ec9b63a8a3ff024"} Jul 09 09:56:07.692759 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:07.692742 2571 scope.go:117] "RemoveContainer" containerID="ff798c018b929a940dc79a8bdd623bb99260f40caa08350d5ec9b63a8a3ff024" Jul 09 09:56:08.698150 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:08.698091 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-bwzqw" event={"ID":"e920c339-7a54-4742-b438-9dddc5ca84a7","Type":"ContainerStarted","Data":"f32f576fc0afc894efbdd84e116c183f304d29067a56502ed1214bb9c299463f"} Jul 09 09:56:11.710444 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:11.710348 2571 generic.go:356] "Generic (PLEG): container finished" podID="90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4" containerID="25e7ad9e7b3bae46175a7200e8b0d4009786b0ffd90bedacdcba909795210862" exitCode=0 Jul 09 09:56:11.710444 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:11.710424 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" event={"ID":"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4","Type":"ContainerDied","Data":"25e7ad9e7b3bae46175a7200e8b0d4009786b0ffd90bedacdcba909795210862"} Jul 09 09:56:11.710905 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:11.710763 2571 scope.go:117] "RemoveContainer" containerID="25e7ad9e7b3bae46175a7200e8b0d4009786b0ffd90bedacdcba909795210862" Jul 09 09:56:12.716531 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:12.716499 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-fpvm6" event={"ID":"90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4","Type":"ContainerStarted","Data":"d042eb51268521c4dde1c7a18c4e33ed041f03162dd044895e29af084d026929"} Jul 09 09:56:13.721494 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:13.721458 2571 generic.go:356] "Generic (PLEG): container finished" podID="a38dd258-2b58-4b2b-ba76-86f8d4f1c474" containerID="612a75d146680f60ab3a65ddac08be78e735e1de6134b176395ac48077ce50a1" exitCode=0 Jul 09 09:56:13.721878 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:13.721492 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" event={"ID":"a38dd258-2b58-4b2b-ba76-86f8d4f1c474","Type":"ContainerDied","Data":"612a75d146680f60ab3a65ddac08be78e735e1de6134b176395ac48077ce50a1"} Jul 09 09:56:13.721939 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:13.721923 2571 scope.go:117] "RemoveContainer" containerID="612a75d146680f60ab3a65ddac08be78e735e1de6134b176395ac48077ce50a1" Jul 09 09:56:14.727374 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:14.727341 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-c5t4m" event={"ID":"a38dd258-2b58-4b2b-ba76-86f8d4f1c474","Type":"ContainerStarted","Data":"f7a0de32a246e816f1fdb9a3634c25f3677b7a03385346c37472e40cc2343762"} Jul 09 09:56:27.745152 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:27.745077 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-f84655c86-v44zh" podUID="82fc126b-f026-4812-80ac-c604395fde81" containerName="console" containerID="cri-o://666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669" gracePeriod=15 Jul 09 09:56:27.981446 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:27.981415 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:56:28.030949 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.030886 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-oauth-serving-cert\") pod \"82fc126b-f026-4812-80ac-c604395fde81\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " Jul 09 09:56:28.030949 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.030940 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-oauth-config\") pod \"82fc126b-f026-4812-80ac-c604395fde81\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " Jul 09 09:56:28.030949 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.030956 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-serving-cert\") pod \"82fc126b-f026-4812-80ac-c604395fde81\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " Jul 09 09:56:28.031215 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.030993 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-console-config\") pod \"82fc126b-f026-4812-80ac-c604395fde81\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " Jul 09 09:56:28.031215 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.031027 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4jtx\" (UniqueName: \"kubernetes.io/projected/82fc126b-f026-4812-80ac-c604395fde81-kube-api-access-s4jtx\") pod \"82fc126b-f026-4812-80ac-c604395fde81\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " Jul 09 09:56:28.031215 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.031046 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-service-ca\") pod \"82fc126b-f026-4812-80ac-c604395fde81\" (UID: \"82fc126b-f026-4812-80ac-c604395fde81\") " Jul 09 09:56:28.031364 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.031331 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "82fc126b-f026-4812-80ac-c604395fde81" (UID: "82fc126b-f026-4812-80ac-c604395fde81"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:56:28.031465 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.031419 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-console-config" (OuterVolumeSpecName: "console-config") pod "82fc126b-f026-4812-80ac-c604395fde81" (UID: "82fc126b-f026-4812-80ac-c604395fde81"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:56:28.031578 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.031501 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-service-ca" (OuterVolumeSpecName: "service-ca") pod "82fc126b-f026-4812-80ac-c604395fde81" (UID: "82fc126b-f026-4812-80ac-c604395fde81"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:56:28.033286 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.033260 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "82fc126b-f026-4812-80ac-c604395fde81" (UID: "82fc126b-f026-4812-80ac-c604395fde81"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:56:28.033527 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.033507 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "82fc126b-f026-4812-80ac-c604395fde81" (UID: "82fc126b-f026-4812-80ac-c604395fde81"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:56:28.033595 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.033540 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fc126b-f026-4812-80ac-c604395fde81-kube-api-access-s4jtx" (OuterVolumeSpecName: "kube-api-access-s4jtx") pod "82fc126b-f026-4812-80ac-c604395fde81" (UID: "82fc126b-f026-4812-80ac-c604395fde81"). InnerVolumeSpecName "kube-api-access-s4jtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:56:28.132100 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.132063 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-oauth-config\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:28.132100 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.132093 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82fc126b-f026-4812-80ac-c604395fde81-console-serving-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:28.132100 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.132104 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-console-config\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:28.132341 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.132113 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4jtx\" (UniqueName: \"kubernetes.io/projected/82fc126b-f026-4812-80ac-c604395fde81-kube-api-access-s4jtx\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:28.132341 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.132140 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:28.132341 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.132148 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82fc126b-f026-4812-80ac-c604395fde81-oauth-serving-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:56:28.779933 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.779899 2571 generic.go:356] "Generic (PLEG): container finished" podID="82fc126b-f026-4812-80ac-c604395fde81" containerID="666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669" exitCode=0 Jul 09 09:56:28.780370 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.779939 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f84655c86-v44zh" event={"ID":"82fc126b-f026-4812-80ac-c604395fde81","Type":"ContainerDied","Data":"666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669"} Jul 09 09:56:28.780370 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.779979 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f84655c86-v44zh" event={"ID":"82fc126b-f026-4812-80ac-c604395fde81","Type":"ContainerDied","Data":"ceff84efd61be1960f28fd1a815b3a76622502fbdfb68edd3cfabf1a13db14eb"} Jul 09 09:56:28.780370 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.779999 2571 scope.go:117] "RemoveContainer" containerID="666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669" Jul 09 09:56:28.780370 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.779943 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f84655c86-v44zh" Jul 09 09:56:28.788821 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.788803 2571 scope.go:117] "RemoveContainer" containerID="666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669" Jul 09 09:56:28.789114 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:56:28.789095 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669\": container with ID starting with 666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669 not found: ID does not exist" containerID="666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669" Jul 09 09:56:28.789178 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.789145 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669"} err="failed to get container status \"666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669\": rpc error: code = NotFound desc = could not find container \"666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669\": container with ID starting with 666b41fbeb81e2c1830e5114907dc60eb57e359829b2771888bba803cc3e2669 not found: ID does not exist" Jul 09 09:56:28.802603 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.802570 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f84655c86-v44zh"] Jul 09 09:56:28.807108 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:28.807082 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f84655c86-v44zh"] Jul 09 09:56:29.159793 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:29.159762 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82fc126b-f026-4812-80ac-c604395fde81" path="/var/lib/kubelet/pods/82fc126b-f026-4812-80ac-c604395fde81/volumes" Jul 09 09:56:57.144455 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.144400 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d85fbc597-gpxsd"] Jul 09 09:56:57.144978 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.144757 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" containerName="registry" Jul 09 09:56:57.144978 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.144773 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" containerName="registry" Jul 09 09:56:57.144978 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.144794 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fc126b-f026-4812-80ac-c604395fde81" containerName="console" Jul 09 09:56:57.144978 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.144799 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fc126b-f026-4812-80ac-c604395fde81" containerName="console" Jul 09 09:56:57.144978 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.144862 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fc126b-f026-4812-80ac-c604395fde81" containerName="console" Jul 09 09:56:57.144978 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.144870 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="57cc59ae-e2bf-4fde-86f2-e1878ae23bb8" containerName="registry" Jul 09 09:56:57.147641 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.147616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.160765 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.160738 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d85fbc597-gpxsd"] Jul 09 09:56:57.259009 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.258922 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-trusted-ca-bundle\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.259205 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.259065 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-oauth-config\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.259205 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.259118 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-config\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.259205 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.259174 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-oauth-serving-cert\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.259372 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.259211 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzzkq\" (UniqueName: \"kubernetes.io/projected/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-kube-api-access-bzzkq\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.259372 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.259283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-serving-cert\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.259372 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.259312 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-service-ca\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.359828 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.359789 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzzkq\" (UniqueName: \"kubernetes.io/projected/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-kube-api-access-bzzkq\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360006 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.359845 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-serving-cert\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360006 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.359957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-service-ca\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360153 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.360007 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-trusted-ca-bundle\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360153 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.360051 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-oauth-config\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360153 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.360083 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-config\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360153 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.360097 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-oauth-serving-cert\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360801 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.360770 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-service-ca\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360908 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.360806 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-config\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360908 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.360842 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-oauth-serving-cert\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.360908 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.360876 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-trusted-ca-bundle\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.362356 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.362339 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-serving-cert\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.362435 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.362399 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-oauth-config\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.369971 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.369952 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzzkq\" (UniqueName: \"kubernetes.io/projected/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-kube-api-access-bzzkq\") pod \"console-6d85fbc597-gpxsd\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.458097 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.458016 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:56:57.587485 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.587458 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d85fbc597-gpxsd"] Jul 09 09:56:57.589804 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:56:57.589781 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0c84ed_49e0_470a_8ccc_a08f2cd5a499.slice/crio-a8283824e7b8c68bf89a18b3cc4316cc45c984aa6a5366d09bd5c6e3cd514344 WatchSource:0}: Error finding container a8283824e7b8c68bf89a18b3cc4316cc45c984aa6a5366d09bd5c6e3cd514344: Status 404 returned error can't find the container with id a8283824e7b8c68bf89a18b3cc4316cc45c984aa6a5366d09bd5c6e3cd514344 Jul 09 09:56:57.893226 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.893188 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d85fbc597-gpxsd" event={"ID":"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499","Type":"ContainerStarted","Data":"e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0"} Jul 09 09:56:57.893423 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.893269 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d85fbc597-gpxsd" event={"ID":"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499","Type":"ContainerStarted","Data":"a8283824e7b8c68bf89a18b3cc4316cc45c984aa6a5366d09bd5c6e3cd514344"} Jul 09 09:56:57.913036 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:56:57.912991 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d85fbc597-gpxsd" podStartSLOduration=0.912976935 podStartE2EDuration="912.976935ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:56:57.912352021 +0000 UTC m=+173.244835819" watchObservedRunningTime="2026-07-09 09:56:57.912976935 +0000 UTC m=+173.245460733" Jul 09 09:57:07.458553 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:07.458514 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:57:07.459055 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:07.458582 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:57:07.462974 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:07.462950 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:57:07.932539 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:07.932511 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 09:57:07.983648 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:07.983617 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86ddb48d55-tqmht"] Jul 09 09:57:33.007102 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.007043 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-86ddb48d55-tqmht" podUID="7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" containerName="console" containerID="cri-o://ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f" gracePeriod=15 Jul 09 09:57:33.251513 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.251489 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:57:33.363700 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.363667 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-service-ca\") pod \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " Jul 09 09:57:33.363886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.363718 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-config\") pod \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " Jul 09 09:57:33.363886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.363746 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-oauth-config\") pod \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " Jul 09 09:57:33.363886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.363764 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-serving-cert\") pod \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " Jul 09 09:57:33.363886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.363793 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-trusted-ca-bundle\") pod \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " Jul 09 09:57:33.363886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.363825 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-oauth-serving-cert\") pod \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " Jul 09 09:57:33.363886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.363843 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djqpx\" (UniqueName: \"kubernetes.io/projected/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-kube-api-access-djqpx\") pod \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\" (UID: \"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb\") " Jul 09 09:57:33.364480 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.364078 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-config" (OuterVolumeSpecName: "console-config") pod "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" (UID: "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:57:33.364480 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.364277 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" (UID: "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:57:33.364480 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.364298 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" (UID: "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:57:33.364480 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.364319 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-service-ca" (OuterVolumeSpecName: "service-ca") pod "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" (UID: "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:57:33.365946 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.365923 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" (UID: "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:57:33.366019 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.365978 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" (UID: "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:57:33.366076 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.366042 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-kube-api-access-djqpx" (OuterVolumeSpecName: "kube-api-access-djqpx") pod "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" (UID: "7a28ac20-26d3-4922-8422-f3fe5cc3e8cb"). InnerVolumeSpecName "kube-api-access-djqpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:57:33.464722 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.464689 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-trusted-ca-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:33.464722 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.464716 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-oauth-serving-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:33.464722 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.464726 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djqpx\" (UniqueName: \"kubernetes.io/projected/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-kube-api-access-djqpx\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:33.464951 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.464735 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:33.464951 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.464745 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-config\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:33.464951 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.464753 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-oauth-config\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:33.464951 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:33.464762 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb-console-serving-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:34.022900 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.022860 2571 generic.go:356] "Generic (PLEG): container finished" podID="7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" containerID="ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f" exitCode=0 Jul 09 09:57:34.023296 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.022909 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86ddb48d55-tqmht" Jul 09 09:57:34.023296 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.022927 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86ddb48d55-tqmht" event={"ID":"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb","Type":"ContainerDied","Data":"ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f"} Jul 09 09:57:34.023296 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.022975 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86ddb48d55-tqmht" event={"ID":"7a28ac20-26d3-4922-8422-f3fe5cc3e8cb","Type":"ContainerDied","Data":"82eab90962545433196826b60205a0272a4cd32e4bf10cb9324cfcc2fc2312d4"} Jul 09 09:57:34.023296 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.023003 2571 scope.go:117] "RemoveContainer" containerID="ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f" Jul 09 09:57:34.031205 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.031187 2571 scope.go:117] "RemoveContainer" containerID="ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f" Jul 09 09:57:34.031459 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:57:34.031438 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f\": container with ID starting with ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f not found: ID does not exist" containerID="ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f" Jul 09 09:57:34.031516 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.031468 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f"} err="failed to get container status \"ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f\": rpc error: code = NotFound desc = could not find container \"ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f\": container with ID starting with ee9891bdb44c546ee725d119d3209322e84c6ac0ae41474caa9a8dbcfde3eb2f not found: ID does not exist" Jul 09 09:57:34.048202 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.048178 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86ddb48d55-tqmht"] Jul 09 09:57:34.054102 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:34.054083 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86ddb48d55-tqmht"] Jul 09 09:57:35.159159 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:35.159109 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" path="/var/lib/kubelet/pods/7a28ac20-26d3-4922-8422-f3fe5cc3e8cb/volumes" Jul 09 09:57:37.302003 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.301971 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb"] Jul 09 09:57:37.302386 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.302250 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" containerName="console" Jul 09 09:57:37.302386 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.302261 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" containerName="console" Jul 09 09:57:37.302386 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.302318 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a28ac20-26d3-4922-8422-f3fe5cc3e8cb" containerName="console" Jul 09 09:57:37.306581 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.306562 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.309560 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.309532 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7q66h\"" Jul 09 09:57:37.309748 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.309731 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 09:57:37.310728 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.310708 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 09:57:37.316410 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.316392 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb"] Jul 09 09:57:37.395641 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.395609 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.395815 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.395653 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cknmz\" (UniqueName: \"kubernetes.io/projected/625adee4-cec9-42ef-a255-6b1aa677ae44-kube-api-access-cknmz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.395815 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.395687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.496373 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.496338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.496525 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.496389 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cknmz\" (UniqueName: \"kubernetes.io/projected/625adee4-cec9-42ef-a255-6b1aa677ae44-kube-api-access-cknmz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.496525 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.496423 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.496724 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.496703 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.496763 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.496726 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.507190 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.507165 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cknmz\" (UniqueName: \"kubernetes.io/projected/625adee4-cec9-42ef-a255-6b1aa677ae44-kube-api-access-cknmz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.616650 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.616612 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:37.746700 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:37.746667 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb"] Jul 09 09:57:37.749037 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:57:37.749010 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625adee4_cec9_42ef_a255_6b1aa677ae44.slice/crio-9dfd01fcaf7a694e6235c35a09afae6e40ea8fff65684461d3e4cd816ce702d5 WatchSource:0}: Error finding container 9dfd01fcaf7a694e6235c35a09afae6e40ea8fff65684461d3e4cd816ce702d5: Status 404 returned error can't find the container with id 9dfd01fcaf7a694e6235c35a09afae6e40ea8fff65684461d3e4cd816ce702d5 Jul 09 09:57:38.039485 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:38.039403 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" event={"ID":"625adee4-cec9-42ef-a255-6b1aa677ae44","Type":"ContainerStarted","Data":"9dfd01fcaf7a694e6235c35a09afae6e40ea8fff65684461d3e4cd816ce702d5"} Jul 09 09:57:44.065550 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:44.065510 2571 generic.go:356] "Generic (PLEG): container finished" podID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerID="2e4b30a4cdcccdfa0902f0d38ccb1ceb7e956c8a5454275a88e10e97e787c776" exitCode=0 Jul 09 09:57:44.066018 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:44.065546 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" event={"ID":"625adee4-cec9-42ef-a255-6b1aa677ae44","Type":"ContainerDied","Data":"2e4b30a4cdcccdfa0902f0d38ccb1ceb7e956c8a5454275a88e10e97e787c776"} Jul 09 09:57:47.077970 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:47.077940 2571 generic.go:356] "Generic (PLEG): container finished" podID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerID="dccf0a78131e13a786d1663411e13305863de3834135d5198af344361baf2422" exitCode=0 Jul 09 09:57:47.078356 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:47.077990 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" event={"ID":"625adee4-cec9-42ef-a255-6b1aa677ae44","Type":"ContainerDied","Data":"dccf0a78131e13a786d1663411e13305863de3834135d5198af344361baf2422"} Jul 09 09:57:55.113933 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:55.113897 2571 generic.go:356] "Generic (PLEG): container finished" podID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerID="785970f9b15007e018743245d8f4e0467dd82d53b502c7ca1d1ee182c1f353ec" exitCode=0 Jul 09 09:57:55.113933 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:55.113926 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" event={"ID":"625adee4-cec9-42ef-a255-6b1aa677ae44","Type":"ContainerDied","Data":"785970f9b15007e018743245d8f4e0467dd82d53b502c7ca1d1ee182c1f353ec"} Jul 09 09:57:56.275195 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.275165 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:56.459456 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.459377 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cknmz\" (UniqueName: \"kubernetes.io/projected/625adee4-cec9-42ef-a255-6b1aa677ae44-kube-api-access-cknmz\") pod \"625adee4-cec9-42ef-a255-6b1aa677ae44\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " Jul 09 09:57:56.459456 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.459434 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-util\") pod \"625adee4-cec9-42ef-a255-6b1aa677ae44\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " Jul 09 09:57:56.459693 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.459464 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-bundle\") pod \"625adee4-cec9-42ef-a255-6b1aa677ae44\" (UID: \"625adee4-cec9-42ef-a255-6b1aa677ae44\") " Jul 09 09:57:56.460178 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.460151 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-bundle" (OuterVolumeSpecName: "bundle") pod "625adee4-cec9-42ef-a255-6b1aa677ae44" (UID: "625adee4-cec9-42ef-a255-6b1aa677ae44"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:57:56.461584 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.461559 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625adee4-cec9-42ef-a255-6b1aa677ae44-kube-api-access-cknmz" (OuterVolumeSpecName: "kube-api-access-cknmz") pod "625adee4-cec9-42ef-a255-6b1aa677ae44" (UID: "625adee4-cec9-42ef-a255-6b1aa677ae44"). InnerVolumeSpecName "kube-api-access-cknmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:57:56.463689 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.463666 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-util" (OuterVolumeSpecName: "util") pod "625adee4-cec9-42ef-a255-6b1aa677ae44" (UID: "625adee4-cec9-42ef-a255-6b1aa677ae44"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:57:56.560952 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.560917 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-util\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:56.560952 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.560943 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/625adee4-cec9-42ef-a255-6b1aa677ae44-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:56.560952 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:56.560953 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cknmz\" (UniqueName: \"kubernetes.io/projected/625adee4-cec9-42ef-a255-6b1aa677ae44-kube-api-access-cknmz\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:57:57.123712 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:57.123686 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" Jul 09 09:57:57.123877 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:57.123713 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540j5tbb" event={"ID":"625adee4-cec9-42ef-a255-6b1aa677ae44","Type":"ContainerDied","Data":"9dfd01fcaf7a694e6235c35a09afae6e40ea8fff65684461d3e4cd816ce702d5"} Jul 09 09:57:57.123877 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:57:57.123758 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dfd01fcaf7a694e6235c35a09afae6e40ea8fff65684461d3e4cd816ce702d5" Jul 09 09:58:00.272228 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.272193 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f"] Jul 09 09:58:00.272675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.272480 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerName="util" Jul 09 09:58:00.272675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.272492 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerName="util" Jul 09 09:58:00.272675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.272505 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerName="extract" Jul 09 09:58:00.272675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.272510 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerName="extract" Jul 09 09:58:00.272675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.272518 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerName="pull" Jul 09 09:58:00.272675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.272524 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerName="pull" Jul 09 09:58:00.272675 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.272572 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="625adee4-cec9-42ef-a255-6b1aa677ae44" containerName="extract" Jul 09 09:58:00.275028 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.275012 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" Jul 09 09:58:00.277811 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.277788 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 09 09:58:00.278279 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.278260 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2x7h2\"" Jul 09 09:58:00.278391 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.278375 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 09 09:58:00.292830 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.292800 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f"] Jul 09 09:58:00.391266 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.391229 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240af6cc-2a77-4a16-82cc-357d11405092-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-czp7f\" (UID: \"240af6cc-2a77-4a16-82cc-357d11405092\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" Jul 09 09:58:00.391266 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.391274 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92ztm\" (UniqueName: \"kubernetes.io/projected/240af6cc-2a77-4a16-82cc-357d11405092-kube-api-access-92ztm\") pod \"cert-manager-operator-controller-manager-55c444bf86-czp7f\" (UID: \"240af6cc-2a77-4a16-82cc-357d11405092\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" Jul 09 09:58:00.492573 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.492538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240af6cc-2a77-4a16-82cc-357d11405092-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-czp7f\" (UID: \"240af6cc-2a77-4a16-82cc-357d11405092\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" Jul 09 09:58:00.492664 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.492580 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92ztm\" (UniqueName: \"kubernetes.io/projected/240af6cc-2a77-4a16-82cc-357d11405092-kube-api-access-92ztm\") pod \"cert-manager-operator-controller-manager-55c444bf86-czp7f\" (UID: \"240af6cc-2a77-4a16-82cc-357d11405092\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" Jul 09 09:58:00.492917 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.492896 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/240af6cc-2a77-4a16-82cc-357d11405092-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-czp7f\" (UID: \"240af6cc-2a77-4a16-82cc-357d11405092\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" Jul 09 09:58:00.519170 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.519136 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92ztm\" (UniqueName: \"kubernetes.io/projected/240af6cc-2a77-4a16-82cc-357d11405092-kube-api-access-92ztm\") pod \"cert-manager-operator-controller-manager-55c444bf86-czp7f\" (UID: \"240af6cc-2a77-4a16-82cc-357d11405092\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" Jul 09 09:58:00.583547 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.583517 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" Jul 09 09:58:00.724148 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:00.724095 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f"] Jul 09 09:58:00.727674 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:58:00.727649 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240af6cc_2a77_4a16_82cc_357d11405092.slice/crio-0f4f3730a75df8cee9b457ebb97e4e80ecf8c8075c0af8e96b0a514ad178920e WatchSource:0}: Error finding container 0f4f3730a75df8cee9b457ebb97e4e80ecf8c8075c0af8e96b0a514ad178920e: Status 404 returned error can't find the container with id 0f4f3730a75df8cee9b457ebb97e4e80ecf8c8075c0af8e96b0a514ad178920e Jul 09 09:58:01.139725 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:01.139689 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" event={"ID":"240af6cc-2a77-4a16-82cc-357d11405092","Type":"ContainerStarted","Data":"0f4f3730a75df8cee9b457ebb97e4e80ecf8c8075c0af8e96b0a514ad178920e"} Jul 09 09:58:04.154192 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.154153 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" event={"ID":"240af6cc-2a77-4a16-82cc-357d11405092","Type":"ContainerStarted","Data":"c09a859304c48536edb13c11353b3e3629d4415ffa9abf1114937bb897c18adb"} Jul 09 09:58:04.193943 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.193887 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-czp7f" podStartSLOduration=1.539719992 podStartE2EDuration="4.193870749s" totalImagesPullingTime="2.654150757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:58:04.19144131 +0000 UTC m=+239.523925108" watchObservedRunningTime="2026-07-09 09:58:04.193870749 +0000 UTC m=+239.526354575" Jul 09 09:58:04.865687 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.865652 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf"] Jul 09 09:58:04.869044 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.869027 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:04.877347 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.877328 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 09:58:04.877535 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.877523 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7q66h\"" Jul 09 09:58:04.877832 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.877819 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 09:58:04.885912 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.885887 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf"] Jul 09 09:58:04.924828 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.924798 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:04.924940 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.924835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:04.924940 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:04.924856 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ts7\" (UniqueName: \"kubernetes.io/projected/c606c702-ea2b-4ac8-b786-be142042d727-kube-api-access-q5ts7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:05.026106 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.026075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:05.026278 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.026117 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ts7\" (UniqueName: \"kubernetes.io/projected/c606c702-ea2b-4ac8-b786-be142042d727-kube-api-access-q5ts7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:05.026278 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.026183 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:05.026495 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.026477 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:05.026538 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.026509 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:05.050977 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.050946 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 09:58:05.060343 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.060325 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 09:58:05.069915 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.069891 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ts7\" (UniqueName: \"kubernetes.io/projected/c606c702-ea2b-4ac8-b786-be142042d727-kube-api-access-q5ts7\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:05.180370 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.180298 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7q66h\"" Jul 09 09:58:05.188687 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.188666 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:05.322365 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.322330 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf"] Jul 09 09:58:05.325857 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:58:05.325830 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc606c702_ea2b_4ac8_b786_be142042d727.slice/crio-03713f235cd61497e5caf9a11ab4f86ff66377f03436792bb8701e3852ce1559 WatchSource:0}: Error finding container 03713f235cd61497e5caf9a11ab4f86ff66377f03436792bb8701e3852ce1559: Status 404 returned error can't find the container with id 03713f235cd61497e5caf9a11ab4f86ff66377f03436792bb8701e3852ce1559 Jul 09 09:58:05.328451 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.328429 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bqdhh"] Jul 09 09:58:05.333827 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.333811 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:05.336792 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.336776 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-qn4v6\"" Jul 09 09:58:05.337072 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.337052 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 09 09:58:05.337257 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.337242 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 09 09:58:05.343326 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.343304 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bqdhh"] Jul 09 09:58:05.428724 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.428685 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6549f361-5039-4087-a2ab-3f2fbfd00778-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bqdhh\" (UID: \"6549f361-5039-4087-a2ab-3f2fbfd00778\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:05.428886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.428725 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg557\" (UniqueName: \"kubernetes.io/projected/6549f361-5039-4087-a2ab-3f2fbfd00778-kube-api-access-hg557\") pod \"cert-manager-webhook-6486ffd997-bqdhh\" (UID: \"6549f361-5039-4087-a2ab-3f2fbfd00778\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:05.529250 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.529216 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6549f361-5039-4087-a2ab-3f2fbfd00778-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bqdhh\" (UID: \"6549f361-5039-4087-a2ab-3f2fbfd00778\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:05.529385 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.529253 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg557\" (UniqueName: \"kubernetes.io/projected/6549f361-5039-4087-a2ab-3f2fbfd00778-kube-api-access-hg557\") pod \"cert-manager-webhook-6486ffd997-bqdhh\" (UID: \"6549f361-5039-4087-a2ab-3f2fbfd00778\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:05.542029 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.541994 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg557\" (UniqueName: \"kubernetes.io/projected/6549f361-5039-4087-a2ab-3f2fbfd00778-kube-api-access-hg557\") pod \"cert-manager-webhook-6486ffd997-bqdhh\" (UID: \"6549f361-5039-4087-a2ab-3f2fbfd00778\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:05.542283 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.542112 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6549f361-5039-4087-a2ab-3f2fbfd00778-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bqdhh\" (UID: \"6549f361-5039-4087-a2ab-3f2fbfd00778\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:05.664840 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.664802 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:05.792425 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:05.792397 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bqdhh"] Jul 09 09:58:05.795661 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:58:05.795635 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6549f361_5039_4087_a2ab_3f2fbfd00778.slice/crio-5fe71c9a5812dec6287539d00774670b756190b48af118e4ec7fa7d5eea0f66d WatchSource:0}: Error finding container 5fe71c9a5812dec6287539d00774670b756190b48af118e4ec7fa7d5eea0f66d: Status 404 returned error can't find the container with id 5fe71c9a5812dec6287539d00774670b756190b48af118e4ec7fa7d5eea0f66d Jul 09 09:58:06.163658 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:06.163620 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" event={"ID":"6549f361-5039-4087-a2ab-3f2fbfd00778","Type":"ContainerStarted","Data":"5fe71c9a5812dec6287539d00774670b756190b48af118e4ec7fa7d5eea0f66d"} Jul 09 09:58:06.165025 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:06.165003 2571 generic.go:356] "Generic (PLEG): container finished" podID="c606c702-ea2b-4ac8-b786-be142042d727" containerID="1418a920f29c6dc9e494f757851369a0e4c01df711017004352221ed5edff607" exitCode=0 Jul 09 09:58:06.165177 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:06.165062 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" event={"ID":"c606c702-ea2b-4ac8-b786-be142042d727","Type":"ContainerDied","Data":"1418a920f29c6dc9e494f757851369a0e4c01df711017004352221ed5edff607"} Jul 09 09:58:06.165177 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:06.165106 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" event={"ID":"c606c702-ea2b-4ac8-b786-be142042d727","Type":"ContainerStarted","Data":"03713f235cd61497e5caf9a11ab4f86ff66377f03436792bb8701e3852ce1559"} Jul 09 09:58:10.183718 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:10.183676 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" event={"ID":"6549f361-5039-4087-a2ab-3f2fbfd00778","Type":"ContainerStarted","Data":"3af5d19014df795a9efdc908f5f3a80eaeff531ec254e454e21e3b87ac3a4d36"} Jul 09 09:58:10.184145 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:10.183791 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:10.185509 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:10.185483 2571 generic.go:356] "Generic (PLEG): container finished" podID="c606c702-ea2b-4ac8-b786-be142042d727" containerID="a31d5c7a5c15a829b49e5cf9d8d182e06b050ea60631826ba3cb09865a473b95" exitCode=0 Jul 09 09:58:10.185633 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:10.185561 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" event={"ID":"c606c702-ea2b-4ac8-b786-be142042d727","Type":"ContainerDied","Data":"a31d5c7a5c15a829b49e5cf9d8d182e06b050ea60631826ba3cb09865a473b95"} Jul 09 09:58:10.203333 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:10.203248 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" podStartSLOduration=1.448562656 podStartE2EDuration="5.203230463s" totalImagesPullingTime="3.754667807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:58:10.203021788 +0000 UTC m=+245.535505588" watchObservedRunningTime="2026-07-09 09:58:10.203230463 +0000 UTC m=+245.535714262" Jul 09 09:58:11.191479 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:11.191442 2571 generic.go:356] "Generic (PLEG): container finished" podID="c606c702-ea2b-4ac8-b786-be142042d727" containerID="f629f6b7267e2805e3a346ed765c9bf9665b42010d29413a65ddbf868ea24924" exitCode=0 Jul 09 09:58:11.191828 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:11.191498 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" event={"ID":"c606c702-ea2b-4ac8-b786-be142042d727","Type":"ContainerDied","Data":"f629f6b7267e2805e3a346ed765c9bf9665b42010d29413a65ddbf868ea24924"} Jul 09 09:58:12.315438 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.315413 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:12.384757 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.384728 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-util\") pod \"c606c702-ea2b-4ac8-b786-be142042d727\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " Jul 09 09:58:12.384886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.384778 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-bundle\") pod \"c606c702-ea2b-4ac8-b786-be142042d727\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " Jul 09 09:58:12.384886 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.384815 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5ts7\" (UniqueName: \"kubernetes.io/projected/c606c702-ea2b-4ac8-b786-be142042d727-kube-api-access-q5ts7\") pod \"c606c702-ea2b-4ac8-b786-be142042d727\" (UID: \"c606c702-ea2b-4ac8-b786-be142042d727\") " Jul 09 09:58:12.385228 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.385202 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-bundle" (OuterVolumeSpecName: "bundle") pod "c606c702-ea2b-4ac8-b786-be142042d727" (UID: "c606c702-ea2b-4ac8-b786-be142042d727"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:58:12.386884 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.386856 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c606c702-ea2b-4ac8-b786-be142042d727-kube-api-access-q5ts7" (OuterVolumeSpecName: "kube-api-access-q5ts7") pod "c606c702-ea2b-4ac8-b786-be142042d727" (UID: "c606c702-ea2b-4ac8-b786-be142042d727"). InnerVolumeSpecName "kube-api-access-q5ts7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:58:12.389523 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.389488 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-util" (OuterVolumeSpecName: "util") pod "c606c702-ea2b-4ac8-b786-be142042d727" (UID: "c606c702-ea2b-4ac8-b786-be142042d727"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:58:12.485986 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.485922 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-util\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:58:12.485986 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.485946 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c606c702-ea2b-4ac8-b786-be142042d727-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:58:12.485986 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:12.485957 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5ts7\" (UniqueName: \"kubernetes.io/projected/c606c702-ea2b-4ac8-b786-be142042d727-kube-api-access-q5ts7\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:58:13.201469 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:13.201440 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" event={"ID":"c606c702-ea2b-4ac8-b786-be142042d727","Type":"ContainerDied","Data":"03713f235cd61497e5caf9a11ab4f86ff66377f03436792bb8701e3852ce1559"} Jul 09 09:58:13.201621 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:13.201488 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03713f235cd61497e5caf9a11ab4f86ff66377f03436792bb8701e3852ce1559" Jul 09 09:58:13.201621 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:13.201448 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvjhzf" Jul 09 09:58:16.193898 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:16.193865 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bqdhh" Jul 09 09:58:24.934387 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.934351 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-x8xdl"] Jul 09 09:58:24.934776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.934671 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c606c702-ea2b-4ac8-b786-be142042d727" containerName="util" Jul 09 09:58:24.934776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.934686 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c606c702-ea2b-4ac8-b786-be142042d727" containerName="util" Jul 09 09:58:24.934776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.934702 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c606c702-ea2b-4ac8-b786-be142042d727" containerName="pull" Jul 09 09:58:24.934776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.934708 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c606c702-ea2b-4ac8-b786-be142042d727" containerName="pull" Jul 09 09:58:24.934776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.934720 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c606c702-ea2b-4ac8-b786-be142042d727" containerName="extract" Jul 09 09:58:24.934776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.934727 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c606c702-ea2b-4ac8-b786-be142042d727" containerName="extract" Jul 09 09:58:24.934947 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.934791 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c606c702-ea2b-4ac8-b786-be142042d727" containerName="extract" Jul 09 09:58:24.938052 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.938035 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-x8xdl" Jul 09 09:58:24.940748 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.940726 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mzlxs\"" Jul 09 09:58:24.948916 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.948896 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-x8xdl"] Jul 09 09:58:24.993163 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.993118 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5aa02e-bef7-4901-8868-0430345f6aed-bound-sa-token\") pod \"cert-manager-55b4d954d5-x8xdl\" (UID: \"1a5aa02e-bef7-4901-8868-0430345f6aed\") " pod="cert-manager/cert-manager-55b4d954d5-x8xdl" Jul 09 09:58:24.993278 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:24.993184 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvr2l\" (UniqueName: \"kubernetes.io/projected/1a5aa02e-bef7-4901-8868-0430345f6aed-kube-api-access-dvr2l\") pod \"cert-manager-55b4d954d5-x8xdl\" (UID: \"1a5aa02e-bef7-4901-8868-0430345f6aed\") " pod="cert-manager/cert-manager-55b4d954d5-x8xdl" Jul 09 09:58:25.093720 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:25.093688 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvr2l\" (UniqueName: \"kubernetes.io/projected/1a5aa02e-bef7-4901-8868-0430345f6aed-kube-api-access-dvr2l\") pod \"cert-manager-55b4d954d5-x8xdl\" (UID: \"1a5aa02e-bef7-4901-8868-0430345f6aed\") " pod="cert-manager/cert-manager-55b4d954d5-x8xdl" Jul 09 09:58:25.093838 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:25.093786 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5aa02e-bef7-4901-8868-0430345f6aed-bound-sa-token\") pod \"cert-manager-55b4d954d5-x8xdl\" (UID: \"1a5aa02e-bef7-4901-8868-0430345f6aed\") " pod="cert-manager/cert-manager-55b4d954d5-x8xdl" Jul 09 09:58:25.102835 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:25.102810 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a5aa02e-bef7-4901-8868-0430345f6aed-bound-sa-token\") pod \"cert-manager-55b4d954d5-x8xdl\" (UID: \"1a5aa02e-bef7-4901-8868-0430345f6aed\") " pod="cert-manager/cert-manager-55b4d954d5-x8xdl" Jul 09 09:58:25.102982 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:25.102963 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvr2l\" (UniqueName: \"kubernetes.io/projected/1a5aa02e-bef7-4901-8868-0430345f6aed-kube-api-access-dvr2l\") pod \"cert-manager-55b4d954d5-x8xdl\" (UID: \"1a5aa02e-bef7-4901-8868-0430345f6aed\") " pod="cert-manager/cert-manager-55b4d954d5-x8xdl" Jul 09 09:58:25.247971 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:25.247869 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-x8xdl" Jul 09 09:58:25.384723 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:25.384700 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-x8xdl"] Jul 09 09:58:25.386706 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:58:25.386673 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a5aa02e_bef7_4901_8868_0430345f6aed.slice/crio-41f2fa4fd01f1b3b30681a9f3d37faf87ce484bbc456fa0e712b32da4cc3fb76 WatchSource:0}: Error finding container 41f2fa4fd01f1b3b30681a9f3d37faf87ce484bbc456fa0e712b32da4cc3fb76: Status 404 returned error can't find the container with id 41f2fa4fd01f1b3b30681a9f3d37faf87ce484bbc456fa0e712b32da4cc3fb76 Jul 09 09:58:26.256814 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:26.256779 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-x8xdl" event={"ID":"1a5aa02e-bef7-4901-8868-0430345f6aed","Type":"ContainerStarted","Data":"666bc2043ff1662de6511f39b9a7ad4a3e3d8d9e8e70f349777706714bf64b32"} Jul 09 09:58:26.256814 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:26.256821 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-x8xdl" event={"ID":"1a5aa02e-bef7-4901-8868-0430345f6aed","Type":"ContainerStarted","Data":"41f2fa4fd01f1b3b30681a9f3d37faf87ce484bbc456fa0e712b32da4cc3fb76"} Jul 09 09:58:26.274408 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:26.274354 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-x8xdl" podStartSLOduration=2.27433655 podStartE2EDuration="2.27433655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:58:26.273651243 +0000 UTC m=+261.606135038" watchObservedRunningTime="2026-07-09 09:58:26.27433655 +0000 UTC m=+261.606820350" Jul 09 09:58:30.839643 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.839565 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r"] Jul 09 09:58:30.841748 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.841730 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:30.847924 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.847901 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 09 09:58:30.848059 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.848018 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 09 09:58:30.849262 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.849232 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7q66h\"" Jul 09 09:58:30.858604 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.858584 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r"] Jul 09 09:58:30.943183 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.943142 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:30.943324 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.943189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:30.943324 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:30.943211 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nq59\" (UniqueName: \"kubernetes.io/projected/b785b6ba-48f0-40ca-950e-e45727264f2e-kube-api-access-7nq59\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:31.044285 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:31.044245 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:31.044285 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:31.044287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:31.044506 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:31.044313 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq59\" (UniqueName: \"kubernetes.io/projected/b785b6ba-48f0-40ca-950e-e45727264f2e-kube-api-access-7nq59\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:31.044642 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:31.044619 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:31.044743 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:31.044647 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:31.058701 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:31.058673 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq59\" (UniqueName: \"kubernetes.io/projected/b785b6ba-48f0-40ca-950e-e45727264f2e-kube-api-access-7nq59\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:31.151638 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:31.151568 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:31.297360 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:31.297321 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r"] Jul 09 09:58:31.300058 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:58:31.300027 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb785b6ba_48f0_40ca_950e_e45727264f2e.slice/crio-bf7bd174a27fd4d80d00ad5d316f0080487ca51b790830b187fc688072388273 WatchSource:0}: Error finding container bf7bd174a27fd4d80d00ad5d316f0080487ca51b790830b187fc688072388273: Status 404 returned error can't find the container with id bf7bd174a27fd4d80d00ad5d316f0080487ca51b790830b187fc688072388273 Jul 09 09:58:32.284531 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:32.284499 2571 generic.go:356] "Generic (PLEG): container finished" podID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerID="2a4f2e120643c5524de846fa50cffecbe9f84e66e3babc40666e03a47af7bae7" exitCode=0 Jul 09 09:58:32.284992 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:32.284550 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" event={"ID":"b785b6ba-48f0-40ca-950e-e45727264f2e","Type":"ContainerDied","Data":"2a4f2e120643c5524de846fa50cffecbe9f84e66e3babc40666e03a47af7bae7"} Jul 09 09:58:32.284992 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:32.284585 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" event={"ID":"b785b6ba-48f0-40ca-950e-e45727264f2e","Type":"ContainerStarted","Data":"bf7bd174a27fd4d80d00ad5d316f0080487ca51b790830b187fc688072388273"} Jul 09 09:58:33.291056 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:33.290977 2571 generic.go:356] "Generic (PLEG): container finished" podID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerID="b50feb858e787a73c9f8fba330584c7a3325883d0d5a54f18efa522ecd106651" exitCode=0 Jul 09 09:58:33.291056 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:33.291041 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" event={"ID":"b785b6ba-48f0-40ca-950e-e45727264f2e","Type":"ContainerDied","Data":"b50feb858e787a73c9f8fba330584c7a3325883d0d5a54f18efa522ecd106651"} Jul 09 09:58:34.297597 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:34.297566 2571 generic.go:356] "Generic (PLEG): container finished" podID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerID="ba66ec0758718cb96bf9fffc2888496cdff734bcaa62aeee8d44d3a4a94b69fc" exitCode=0 Jul 09 09:58:34.297980 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:34.297631 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" event={"ID":"b785b6ba-48f0-40ca-950e-e45727264f2e","Type":"ContainerDied","Data":"ba66ec0758718cb96bf9fffc2888496cdff734bcaa62aeee8d44d3a4a94b69fc"} Jul 09 09:58:35.431897 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.431871 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:35.582488 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.582452 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-bundle\") pod \"b785b6ba-48f0-40ca-950e-e45727264f2e\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " Jul 09 09:58:35.582657 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.582506 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-util\") pod \"b785b6ba-48f0-40ca-950e-e45727264f2e\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " Jul 09 09:58:35.582657 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.582542 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nq59\" (UniqueName: \"kubernetes.io/projected/b785b6ba-48f0-40ca-950e-e45727264f2e-kube-api-access-7nq59\") pod \"b785b6ba-48f0-40ca-950e-e45727264f2e\" (UID: \"b785b6ba-48f0-40ca-950e-e45727264f2e\") " Jul 09 09:58:35.583255 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.583226 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-bundle" (OuterVolumeSpecName: "bundle") pod "b785b6ba-48f0-40ca-950e-e45727264f2e" (UID: "b785b6ba-48f0-40ca-950e-e45727264f2e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:58:35.584672 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.584650 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b785b6ba-48f0-40ca-950e-e45727264f2e-kube-api-access-7nq59" (OuterVolumeSpecName: "kube-api-access-7nq59") pod "b785b6ba-48f0-40ca-950e-e45727264f2e" (UID: "b785b6ba-48f0-40ca-950e-e45727264f2e"). InnerVolumeSpecName "kube-api-access-7nq59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:58:35.588174 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.588150 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-util" (OuterVolumeSpecName: "util") pod "b785b6ba-48f0-40ca-950e-e45727264f2e" (UID: "b785b6ba-48f0-40ca-950e-e45727264f2e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:58:35.683410 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.683384 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:58:35.683410 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.683405 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b785b6ba-48f0-40ca-950e-e45727264f2e-util\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:58:35.683410 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:35.683414 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nq59\" (UniqueName: \"kubernetes.io/projected/b785b6ba-48f0-40ca-950e-e45727264f2e-kube-api-access-7nq59\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:58:36.307939 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:36.307913 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" Jul 09 09:58:36.308093 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:36.307940 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af6qg6r" event={"ID":"b785b6ba-48f0-40ca-950e-e45727264f2e","Type":"ContainerDied","Data":"bf7bd174a27fd4d80d00ad5d316f0080487ca51b790830b187fc688072388273"} Jul 09 09:58:36.308093 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:36.307985 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7bd174a27fd4d80d00ad5d316f0080487ca51b790830b187fc688072388273" Jul 09 09:58:47.315711 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.315678 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp"] Jul 09 09:58:47.316089 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.315947 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerName="pull" Jul 09 09:58:47.316089 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.315958 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerName="pull" Jul 09 09:58:47.316089 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.315965 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerName="extract" Jul 09 09:58:47.316089 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.315970 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerName="extract" Jul 09 09:58:47.316089 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.315984 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerName="util" Jul 09 09:58:47.316089 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.315989 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerName="util" Jul 09 09:58:47.316089 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.316041 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b785b6ba-48f0-40ca-950e-e45727264f2e" containerName="extract" Jul 09 09:58:47.322490 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.322468 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.326582 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.326558 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 09:58:47.327200 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.327182 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 09:58:47.327274 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.327254 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zsbc7\"" Jul 09 09:58:47.329454 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.329218 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 09:58:47.333681 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.333639 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 09:58:47.351897 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.351874 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp"] Jul 09 09:58:47.484768 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.484741 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a70bdda0-c27f-4934-912a-9e2a4de490bb-manifests\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.484916 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.484779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a70bdda0-c27f-4934-912a-9e2a4de490bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.484916 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.484799 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a70bdda0-c27f-4934-912a-9e2a4de490bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.484916 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.484822 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgql5\" (UniqueName: \"kubernetes.io/projected/a70bdda0-c27f-4934-912a-9e2a4de490bb-kube-api-access-tgql5\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.485026 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.484920 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70bdda0-c27f-4934-912a-9e2a4de490bb-tmp\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.585868 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.585786 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a70bdda0-c27f-4934-912a-9e2a4de490bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.585868 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.585820 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a70bdda0-c27f-4934-912a-9e2a4de490bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.585868 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.585838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgql5\" (UniqueName: \"kubernetes.io/projected/a70bdda0-c27f-4934-912a-9e2a4de490bb-kube-api-access-tgql5\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.586151 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.585883 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70bdda0-c27f-4934-912a-9e2a4de490bb-tmp\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.586151 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.585914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a70bdda0-c27f-4934-912a-9e2a4de490bb-manifests\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.586273 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.586243 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a70bdda0-c27f-4934-912a-9e2a4de490bb-manifests\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.588238 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.588208 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70bdda0-c27f-4934-912a-9e2a4de490bb-tmp\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.588562 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.588540 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a70bdda0-c27f-4934-912a-9e2a4de490bb-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.588724 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.588703 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a70bdda0-c27f-4934-912a-9e2a4de490bb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.615922 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.615897 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgql5\" (UniqueName: \"kubernetes.io/projected/a70bdda0-c27f-4934-912a-9e2a4de490bb-kube-api-access-tgql5\") pod \"opendatahub-operator-controller-manager-5cb598dddb-lccbp\" (UID: \"a70bdda0-c27f-4934-912a-9e2a4de490bb\") " pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.632753 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.632728 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:47.790776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:47.790752 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp"] Jul 09 09:58:47.793299 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:58:47.793267 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70bdda0_c27f_4934_912a_9e2a4de490bb.slice/crio-7bd2081aece05cd44158b0703d96377e487842742c2204c303897b743b2ae06e WatchSource:0}: Error finding container 7bd2081aece05cd44158b0703d96377e487842742c2204c303897b743b2ae06e: Status 404 returned error can't find the container with id 7bd2081aece05cd44158b0703d96377e487842742c2204c303897b743b2ae06e Jul 09 09:58:48.360245 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:48.360214 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" event={"ID":"a70bdda0-c27f-4934-912a-9e2a4de490bb","Type":"ContainerStarted","Data":"7bd2081aece05cd44158b0703d96377e487842742c2204c303897b743b2ae06e"} Jul 09 09:58:51.381331 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:51.381288 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" event={"ID":"a70bdda0-c27f-4934-912a-9e2a4de490bb","Type":"ContainerStarted","Data":"890db21a4b8a1546898c4c3d69da20edc915be750e7ab523c25ed27910d14009"} Jul 09 09:58:52.387638 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:52.387549 2571 generic.go:356] "Generic (PLEG): container finished" podID="a70bdda0-c27f-4934-912a-9e2a4de490bb" containerID="890db21a4b8a1546898c4c3d69da20edc915be750e7ab523c25ed27910d14009" exitCode=0 Jul 09 09:58:52.387977 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:52.387640 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" event={"ID":"a70bdda0-c27f-4934-912a-9e2a4de490bb","Type":"ContainerDied","Data":"890db21a4b8a1546898c4c3d69da20edc915be750e7ab523c25ed27910d14009"} Jul 09 09:58:53.393825 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:53.393791 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" event={"ID":"a70bdda0-c27f-4934-912a-9e2a4de490bb","Type":"ContainerStarted","Data":"c3c7541e605af8591fd19eefeb5f37a9803316a992b5a3df48c3d7062b5f967a"} Jul 09 09:58:53.394447 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:53.393895 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:58:53.437315 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:58:53.437267 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" podStartSLOduration=1.791589486 podStartE2EDuration="6.437251051s" totalImagesPullingTime="3.564028878s" totalInitContainerRuntime="1.081632687s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:58:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:58:53.436829564 +0000 UTC m=+288.769313361" watchObservedRunningTime="2026-07-09 09:58:53.437251051 +0000 UTC m=+288.769734847" Jul 09 09:59:04.400942 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:04.400911 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cb598dddb-lccbp" Jul 09 09:59:05.036331 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:05.036299 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 09:59:05.037087 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:05.037067 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 09:59:05.053023 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:05.052999 2571 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 09:59:09.347469 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.347426 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-589b555f76-zvd84"] Jul 09 09:59:09.353533 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.353466 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.364361 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.364337 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 09:59:09.364543 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.364525 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 09:59:09.468657 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.468622 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bea99a-6583-42ac-b649-d187076552e1-tls-certs\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.468814 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.468741 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0bea99a-6583-42ac-b649-d187076552e1-tmp\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.468814 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.468789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjj4q\" (UniqueName: \"kubernetes.io/projected/c0bea99a-6583-42ac-b649-d187076552e1-kube-api-access-vjj4q\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.494189 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.494156 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-589b555f76-zvd84"] Jul 09 09:59:09.570158 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.570092 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bea99a-6583-42ac-b649-d187076552e1-tls-certs\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.570362 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.570219 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0bea99a-6583-42ac-b649-d187076552e1-tmp\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.570362 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.570264 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjj4q\" (UniqueName: \"kubernetes.io/projected/c0bea99a-6583-42ac-b649-d187076552e1-kube-api-access-vjj4q\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.572605 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.572572 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c0bea99a-6583-42ac-b649-d187076552e1-tmp\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.572758 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.572737 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0bea99a-6583-42ac-b649-d187076552e1-tls-certs\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.609496 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.609432 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjj4q\" (UniqueName: \"kubernetes.io/projected/c0bea99a-6583-42ac-b649-d187076552e1-kube-api-access-vjj4q\") pod \"kube-auth-proxy-589b555f76-zvd84\" (UID: \"c0bea99a-6583-42ac-b649-d187076552e1\") " pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.663933 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.663901 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" Jul 09 09:59:09.683150 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.683110 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-krxns"] Jul 09 09:59:09.686761 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.686741 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:09.694907 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:09.694865 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"odh-model-controller-7998c67dc4-krxns\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="2558c0ba-ad0a-4c79-9ec3-01d4a99130f2" pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:09.695152 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:09.695101 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" Jul 09 09:59:09.695213 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:09.695149 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-dockercfg-68kff\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-68kff\"" type="*v1.Secret" Jul 09 09:59:09.710658 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.710628 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-krxns"] Jul 09 09:59:09.772288 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.772249 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78cr\" (UniqueName: \"kubernetes.io/projected/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-kube-api-access-r78cr\") pod \"odh-model-controller-7998c67dc4-krxns\" (UID: \"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:09.772427 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.772355 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-cert\") pod \"odh-model-controller-7998c67dc4-krxns\" (UID: \"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:09.830045 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.830013 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-589b555f76-zvd84"] Jul 09 09:59:09.830205 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:59:09.830165 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bea99a_6583_42ac_b649_d187076552e1.slice/crio-876c7992420a9ff5cfd4802cf673f84ea0927172fc8787ddd2e0d02f7a1979a2 WatchSource:0}: Error finding container 876c7992420a9ff5cfd4802cf673f84ea0927172fc8787ddd2e0d02f7a1979a2: Status 404 returned error can't find the container with id 876c7992420a9ff5cfd4802cf673f84ea0927172fc8787ddd2e0d02f7a1979a2 Jul 09 09:59:09.832086 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.832069 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 09:59:09.873169 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.873079 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-cert\") pod \"odh-model-controller-7998c67dc4-krxns\" (UID: \"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:09.873169 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.873142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78cr\" (UniqueName: \"kubernetes.io/projected/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-kube-api-access-r78cr\") pod \"odh-model-controller-7998c67dc4-krxns\" (UID: \"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:09.887404 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:09.887375 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78cr\" (UniqueName: \"kubernetes.io/projected/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-kube-api-access-r78cr\") pod \"odh-model-controller-7998c67dc4-krxns\" (UID: \"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:10.474985 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:10.474945 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" event={"ID":"c0bea99a-6583-42ac-b649-d187076552e1","Type":"ContainerStarted","Data":"876c7992420a9ff5cfd4802cf673f84ea0927172fc8787ddd2e0d02f7a1979a2"} Jul 09 09:59:10.641306 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:10.641269 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-68kff\"" Jul 09 09:59:10.873451 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:10.873419 2571 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: failed to sync secret cache: timed out waiting for the condition Jul 09 09:59:10.873619 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:10.873510 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-cert podName:2558c0ba-ad0a-4c79-9ec3-01d4a99130f2 nodeName:}" failed. No retries permitted until 2026-07-09 09:59:11.373492855 +0000 UTC m=+306.705976630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-cert") pod "odh-model-controller-7998c67dc4-krxns" (UID: "2558c0ba-ad0a-4c79-9ec3-01d4a99130f2") : failed to sync secret cache: timed out waiting for the condition Jul 09 09:59:11.158143 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:11.158041 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 09 09:59:11.388355 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:11.388315 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-cert\") pod \"odh-model-controller-7998c67dc4-krxns\" (UID: \"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:11.391298 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:11.391268 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2558c0ba-ad0a-4c79-9ec3-01d4a99130f2-cert\") pod \"odh-model-controller-7998c67dc4-krxns\" (UID: \"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2\") " pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:11.497947 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:11.497872 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:11.676482 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:11.676448 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-krxns"] Jul 09 09:59:11.696848 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:59:11.696818 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2558c0ba_ad0a_4c79_9ec3_01d4a99130f2.slice/crio-bab6632f49b12d5da566a20bb9016d6764e08cafd9c21f6e93210635fc0e1bc5 WatchSource:0}: Error finding container bab6632f49b12d5da566a20bb9016d6764e08cafd9c21f6e93210635fc0e1bc5: Status 404 returned error can't find the container with id bab6632f49b12d5da566a20bb9016d6764e08cafd9c21f6e93210635fc0e1bc5 Jul 09 09:59:12.219453 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.219420 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h6njl"] Jul 09 09:59:12.223765 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.223742 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.227916 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.227887 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 09:59:12.228611 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.228591 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 09 09:59:12.228904 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.228892 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 09 09:59:12.233144 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.233099 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rlxbb\"" Jul 09 09:59:12.233306 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.233284 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 09 09:59:12.254874 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.254838 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h6njl"] Jul 09 09:59:12.297292 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.297263 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01968f5a-a416-45c1-b1cb-f360f0fba0e3-local-certs\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.297292 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.297293 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-token\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.297493 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.297314 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh88c\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-kube-api-access-wh88c\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.297493 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.297367 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.297493 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.297429 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-cacerts\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.297493 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.297453 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.297661 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.297503 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.398809 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.398773 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01968f5a-a416-45c1-b1cb-f360f0fba0e3-local-certs\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.398809 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.398814 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-token\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.399152 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.398838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh88c\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-kube-api-access-wh88c\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.399152 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.398894 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.399152 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.398974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-cacerts\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.399152 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.399015 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.399152 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.399059 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.399557 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.399526 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.401227 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.401186 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01968f5a-a416-45c1-b1cb-f360f0fba0e3-local-certs\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.401440 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.401422 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.401726 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.401709 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-cacerts\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.401874 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.401839 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.413984 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.413963 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-token\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.414111 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.414020 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh88c\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-kube-api-access-wh88c\") pod \"istiod-openshift-gateway-7574459656-h6njl\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:12.487318 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.487247 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-krxns" event={"ID":"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2","Type":"ContainerStarted","Data":"bab6632f49b12d5da566a20bb9016d6764e08cafd9c21f6e93210635fc0e1bc5"} Jul 09 09:59:12.535692 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:12.535464 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:14.935119 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:14.935094 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h6njl"] Jul 09 09:59:14.938535 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:59:14.938489 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01968f5a_a416_45c1_b1cb_f360f0fba0e3.slice/crio-0b1df13ebc9218f1f66779021e876fc0f5de755bf722579fa3ab1dee7e0c71d8 WatchSource:0}: Error finding container 0b1df13ebc9218f1f66779021e876fc0f5de755bf722579fa3ab1dee7e0c71d8: Status 404 returned error can't find the container with id 0b1df13ebc9218f1f66779021e876fc0f5de755bf722579fa3ab1dee7e0c71d8 Jul 09 09:59:15.505065 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:15.505023 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" event={"ID":"c0bea99a-6583-42ac-b649-d187076552e1","Type":"ContainerStarted","Data":"79a586f87ea9e3cec5d7454f23697f05343408f482affcc53f29b1197c5ab156"} Jul 09 09:59:15.506399 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:15.506372 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-krxns" event={"ID":"2558c0ba-ad0a-4c79-9ec3-01d4a99130f2","Type":"ContainerStarted","Data":"79e40aa8fb305db575d441719e56690b8d6ed8891bbffee499b5e7f45ec247c7"} Jul 09 09:59:15.506527 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:15.506435 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:15.507518 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:15.507499 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" event={"ID":"01968f5a-a416-45c1-b1cb-f360f0fba0e3","Type":"ContainerStarted","Data":"0b1df13ebc9218f1f66779021e876fc0f5de755bf722579fa3ab1dee7e0c71d8"} Jul 09 09:59:15.541022 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:15.540978 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-589b555f76-zvd84" podStartSLOduration=1.600785353 podStartE2EDuration="6.540963809s" totalImagesPullingTime="4.940178456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:59:15.540948831 +0000 UTC m=+310.873432628" watchObservedRunningTime="2026-07-09 09:59:15.540963809 +0000 UTC m=+310.873447607" Jul 09 09:59:15.589826 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:15.589775 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-krxns" podStartSLOduration=3.454166955 podStartE2EDuration="6.589757743s" totalImagesPullingTime="3.135590788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:59:15.588404589 +0000 UTC m=+310.920888386" watchObservedRunningTime="2026-07-09 09:59:15.589757743 +0000 UTC m=+310.922241541" Jul 09 09:59:17.577758 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:17.577694 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 09:59:17.578203 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:17.577792 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 09:59:18.524731 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:18.524696 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" event={"ID":"01968f5a-a416-45c1-b1cb-f360f0fba0e3","Type":"ContainerStarted","Data":"b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1"} Jul 09 09:59:18.524960 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:18.524895 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:18.526927 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:18.526885 2571 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-h6njl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 09:59:18.527390 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:18.527356 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" podUID="01968f5a-a416-45c1-b1cb-f360f0fba0e3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 09:59:18.574887 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:18.574834 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" podStartSLOduration=3.938569937 podStartE2EDuration="6.574820424s" totalImagesPullingTime="2.636250487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:59:18.573466513 +0000 UTC m=+313.905950310" watchObservedRunningTime="2026-07-09 09:59:18.574820424 +0000 UTC m=+313.907304222" Jul 09 09:59:19.531530 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:19.531501 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:26.515571 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:26.515545 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-krxns" Jul 09 09:59:27.799002 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:27.798955 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6lq29"] Jul 09 09:59:27.802180 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:27.802163 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:27.848326 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:27.848260 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kserve-controller-manager-d6f4b65b8-6lq29\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="7c522cf7-aa1e-4e1a-82fb-2571a842c8c5" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:27.848326 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:27.848299 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-controller-manager-dockercfg-sgjfn\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-sgjfn\"" type="*v1.Secret" Jul 09 09:59:27.848505 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:27.848351 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kserve-webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" Jul 09 09:59:27.946539 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:27.946507 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-cert\") pod \"kserve-controller-manager-d6f4b65b8-6lq29\" (UID: \"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:27.946539 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:27.946543 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpxzq\" (UniqueName: \"kubernetes.io/projected/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-kube-api-access-zpxzq\") pod \"kserve-controller-manager-d6f4b65b8-6lq29\" (UID: \"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:28.011500 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.011463 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p"] Jul 09 09:59:28.014614 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.014598 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:28.023789 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.023764 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6lq29"] Jul 09 09:59:28.030682 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.030659 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 09:59:28.030789 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.030659 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-92f6r\"" Jul 09 09:59:28.047742 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.047718 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpxzq\" (UniqueName: \"kubernetes.io/projected/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-kube-api-access-zpxzq\") pod \"kserve-controller-manager-d6f4b65b8-6lq29\" (UID: \"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:28.047865 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.047831 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-cert\") pod \"kserve-controller-manager-d6f4b65b8-6lq29\" (UID: \"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:28.061196 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.061177 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpxzq\" (UniqueName: \"kubernetes.io/projected/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-kube-api-access-zpxzq\") pod \"kserve-controller-manager-d6f4b65b8-6lq29\" (UID: \"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:28.096754 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.096724 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p"] Jul 09 09:59:28.148315 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.148280 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqqb\" (UniqueName: \"kubernetes.io/projected/bec5ef67-8c48-44d6-9103-d73e75fa3d82-kube-api-access-4sqqb\") pod \"llmisvc-controller-manager-777c6bd7d5-47n9p\" (UID: \"bec5ef67-8c48-44d6-9103-d73e75fa3d82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:28.148461 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.148397 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec5ef67-8c48-44d6-9103-d73e75fa3d82-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-47n9p\" (UID: \"bec5ef67-8c48-44d6-9103-d73e75fa3d82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:28.249170 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.249116 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec5ef67-8c48-44d6-9103-d73e75fa3d82-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-47n9p\" (UID: \"bec5ef67-8c48-44d6-9103-d73e75fa3d82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:28.249323 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.249203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqqb\" (UniqueName: \"kubernetes.io/projected/bec5ef67-8c48-44d6-9103-d73e75fa3d82-kube-api-access-4sqqb\") pod \"llmisvc-controller-manager-777c6bd7d5-47n9p\" (UID: \"bec5ef67-8c48-44d6-9103-d73e75fa3d82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:28.251656 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.251625 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec5ef67-8c48-44d6-9103-d73e75fa3d82-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-47n9p\" (UID: \"bec5ef67-8c48-44d6-9103-d73e75fa3d82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:28.264048 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.264025 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqqb\" (UniqueName: \"kubernetes.io/projected/bec5ef67-8c48-44d6-9103-d73e75fa3d82-kube-api-access-4sqqb\") pod \"llmisvc-controller-manager-777c6bd7d5-47n9p\" (UID: \"bec5ef67-8c48-44d6-9103-d73e75fa3d82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:28.324465 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.324403 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:28.471717 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.468243 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p"] Jul 09 09:59:28.571882 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:28.571845 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" event={"ID":"bec5ef67-8c48-44d6-9103-d73e75fa3d82","Type":"ContainerStarted","Data":"ea0d9233a90bf8e73ac06b6fa1f31b02b439974341f7f53adfda0ef2bdbc23b1"} Jul 09 09:59:29.048872 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:29.048839 2571 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: failed to sync secret cache: timed out waiting for the condition Jul 09 09:59:29.049263 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:29.048933 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-cert podName:7c522cf7-aa1e-4e1a-82fb-2571a842c8c5 nodeName:}" failed. No retries permitted until 2026-07-09 09:59:29.548914564 +0000 UTC m=+324.881398339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-cert") pod "kserve-controller-manager-d6f4b65b8-6lq29" (UID: "7c522cf7-aa1e-4e1a-82fb-2571a842c8c5") : failed to sync secret cache: timed out waiting for the condition Jul 09 09:59:29.127178 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:29.127153 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-sgjfn\"" Jul 09 09:59:29.295763 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:29.295729 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 09 09:59:29.561414 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:29.561385 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-cert\") pod \"kserve-controller-manager-d6f4b65b8-6lq29\" (UID: \"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:29.563719 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:29.563691 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c522cf7-aa1e-4e1a-82fb-2571a842c8c5-cert\") pod \"kserve-controller-manager-d6f4b65b8-6lq29\" (UID: \"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:29.611735 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:29.611706 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:29.780533 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:29.780497 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-6lq29"] Jul 09 09:59:29.849621 ip-10-0-140-69 kubenswrapper[2571]: W0709 09:59:29.849555 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c522cf7_aa1e_4e1a_82fb_2571a842c8c5.slice/crio-df24caefc6f103a3caa454ae7791586e1387a10e68402387fe959580f358b998 WatchSource:0}: Error finding container df24caefc6f103a3caa454ae7791586e1387a10e68402387fe959580f358b998: Status 404 returned error can't find the container with id df24caefc6f103a3caa454ae7791586e1387a10e68402387fe959580f358b998 Jul 09 09:59:29.954639 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:29.953848 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h6njl"] Jul 09 09:59:29.954639 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:29.954085 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" podUID="01968f5a-a416-45c1-b1cb-f360f0fba0e3" containerName="discovery" containerID="cri-o://b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1" gracePeriod=30 Jul 09 09:59:30.267210 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.267177 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:30.370887 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.370852 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-token\") pod \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " Jul 09 09:59:30.371045 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.370909 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-kubeconfig\") pod \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " Jul 09 09:59:30.371045 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.370939 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01968f5a-a416-45c1-b1cb-f360f0fba0e3-local-certs\") pod \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " Jul 09 09:59:30.371045 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.371013 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-dns-cert\") pod \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " Jul 09 09:59:30.371297 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.371055 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-cacerts\") pod \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " Jul 09 09:59:30.371297 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.371090 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-ca-configmap\") pod \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " Jul 09 09:59:30.371297 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.371172 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh88c\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-kube-api-access-wh88c\") pod \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\" (UID: \"01968f5a-a416-45c1-b1cb-f360f0fba0e3\") " Jul 09 09:59:30.371939 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.371855 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "01968f5a-a416-45c1-b1cb-f360f0fba0e3" (UID: "01968f5a-a416-45c1-b1cb-f360f0fba0e3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 09:59:30.374363 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.374336 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "01968f5a-a416-45c1-b1cb-f360f0fba0e3" (UID: "01968f5a-a416-45c1-b1cb-f360f0fba0e3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:59:30.374565 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.374541 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "01968f5a-a416-45c1-b1cb-f360f0fba0e3" (UID: "01968f5a-a416-45c1-b1cb-f360f0fba0e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:59:30.374650 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.374605 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "01968f5a-a416-45c1-b1cb-f360f0fba0e3" (UID: "01968f5a-a416-45c1-b1cb-f360f0fba0e3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:59:30.374890 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.374849 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-cacerts" (OuterVolumeSpecName: "cacerts") pod "01968f5a-a416-45c1-b1cb-f360f0fba0e3" (UID: "01968f5a-a416-45c1-b1cb-f360f0fba0e3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 09:59:30.375590 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.375561 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-kube-api-access-wh88c" (OuterVolumeSpecName: "kube-api-access-wh88c") pod "01968f5a-a416-45c1-b1cb-f360f0fba0e3" (UID: "01968f5a-a416-45c1-b1cb-f360f0fba0e3"). InnerVolumeSpecName "kube-api-access-wh88c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 09:59:30.375994 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.375905 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01968f5a-a416-45c1-b1cb-f360f0fba0e3-local-certs" (OuterVolumeSpecName: "local-certs") pod "01968f5a-a416-45c1-b1cb-f360f0fba0e3" (UID: "01968f5a-a416-45c1-b1cb-f360f0fba0e3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 09:59:30.472776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.472691 2571 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-cacerts\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:59:30.472776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.472727 2571 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-ca-configmap\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:59:30.472776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.472743 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh88c\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-kube-api-access-wh88c\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:59:30.472776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.472756 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-token\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:59:30.472776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.472768 2571 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-kubeconfig\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:59:30.472776 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.472779 2571 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/01968f5a-a416-45c1-b1cb-f360f0fba0e3-local-certs\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:59:30.473092 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.472790 2571 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/01968f5a-a416-45c1-b1cb-f360f0fba0e3-istio-csr-dns-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 09:59:30.583668 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.583629 2571 generic.go:356] "Generic (PLEG): container finished" podID="01968f5a-a416-45c1-b1cb-f360f0fba0e3" containerID="b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1" exitCode=0 Jul 09 09:59:30.583856 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.583696 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" Jul 09 09:59:30.583856 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.583728 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" event={"ID":"01968f5a-a416-45c1-b1cb-f360f0fba0e3","Type":"ContainerDied","Data":"b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1"} Jul 09 09:59:30.583856 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.583797 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-h6njl" event={"ID":"01968f5a-a416-45c1-b1cb-f360f0fba0e3","Type":"ContainerDied","Data":"0b1df13ebc9218f1f66779021e876fc0f5de755bf722579fa3ab1dee7e0c71d8"} Jul 09 09:59:30.583856 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.583824 2571 scope.go:117] "RemoveContainer" containerID="b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1" Jul 09 09:59:30.585040 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.585011 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" event={"ID":"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5","Type":"ContainerStarted","Data":"df24caefc6f103a3caa454ae7791586e1387a10e68402387fe959580f358b998"} Jul 09 09:59:30.593871 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.593855 2571 scope.go:117] "RemoveContainer" containerID="b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1" Jul 09 09:59:30.594106 ip-10-0-140-69 kubenswrapper[2571]: E0709 09:59:30.594085 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1\": container with ID starting with b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1 not found: ID does not exist" containerID="b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1" Jul 09 09:59:30.594228 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.594118 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1"} err="failed to get container status \"b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1\": rpc error: code = NotFound desc = could not find container \"b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1\": container with ID starting with b3e73f622b20effa24b0d6661d9ab94c5ed8d31147259026dcb07496e4575da1 not found: ID does not exist" Jul 09 09:59:30.635491 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.635463 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h6njl"] Jul 09 09:59:30.663501 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:30.663465 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-h6njl"] Jul 09 09:59:31.162044 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:31.162011 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01968f5a-a416-45c1-b1cb-f360f0fba0e3" path="/var/lib/kubelet/pods/01968f5a-a416-45c1-b1cb-f360f0fba0e3/volumes" Jul 09 09:59:32.597780 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:32.597744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" event={"ID":"bec5ef67-8c48-44d6-9103-d73e75fa3d82","Type":"ContainerStarted","Data":"98a56e086360fdb0dcd44ee6a4717c363655ecff2716741df94510157a8eaf64"} Jul 09 09:59:32.598257 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:32.597895 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 09:59:32.599288 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:32.599263 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" event={"ID":"7c522cf7-aa1e-4e1a-82fb-2571a842c8c5","Type":"ContainerStarted","Data":"147f6aa7de10767e48c8fa10c45962ce43b88df1b4bac74154edc9af487f1cef"} Jul 09 09:59:32.599406 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:32.599308 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 09:59:32.718009 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:32.717907 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" podStartSLOduration=2.565255271 podStartE2EDuration="5.717892895s" totalImagesPullingTime="3.152637624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:59:32.71559092 +0000 UTC m=+328.048074718" watchObservedRunningTime="2026-07-09 09:59:32.717892895 +0000 UTC m=+328.050376715" Jul 09 09:59:32.787171 ip-10-0-140-69 kubenswrapper[2571]: I0709 09:59:32.787099 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" podStartSLOduration=3.25133552 podStartE2EDuration="5.787083742s" totalImagesPullingTime="2.535748222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:59:32.779332937 +0000 UTC m=+328.111816734" watchObservedRunningTime="2026-07-09 09:59:32.787083742 +0000 UTC m=+328.119567523" Jul 09 10:00:03.606274 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:03.606195 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47n9p" Jul 09 10:00:03.609318 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:03.609296 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-6lq29" Jul 09 10:00:10.005035 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.004997 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8pr2c"] Jul 09 10:00:10.005563 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.005497 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01968f5a-a416-45c1-b1cb-f360f0fba0e3" containerName="discovery" Jul 09 10:00:10.005563 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.005513 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="01968f5a-a416-45c1-b1cb-f360f0fba0e3" containerName="discovery" Jul 09 10:00:10.005667 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.005576 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="01968f5a-a416-45c1-b1cb-f360f0fba0e3" containerName="discovery" Jul 09 10:00:10.012251 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.012222 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" Jul 09 10:00:10.019573 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.019545 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 10:00:10.019941 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.019641 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 10:00:10.020467 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.020440 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-9d772\"" Jul 09 10:00:10.025910 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.025886 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8pr2c"] Jul 09 10:00:10.120665 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.120622 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwbwv\" (UniqueName: \"kubernetes.io/projected/c55d597d-f42e-47c9-9825-3188bea3a6cb-kube-api-access-qwbwv\") pod \"kuadrant-operator-catalog-8pr2c\" (UID: \"c55d597d-f42e-47c9-9825-3188bea3a6cb\") " pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" Jul 09 10:00:10.222109 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.222070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbwv\" (UniqueName: \"kubernetes.io/projected/c55d597d-f42e-47c9-9825-3188bea3a6cb-kube-api-access-qwbwv\") pod \"kuadrant-operator-catalog-8pr2c\" (UID: \"c55d597d-f42e-47c9-9825-3188bea3a6cb\") " pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" Jul 09 10:00:10.240591 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.240553 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbwv\" (UniqueName: \"kubernetes.io/projected/c55d597d-f42e-47c9-9825-3188bea3a6cb-kube-api-access-qwbwv\") pod \"kuadrant-operator-catalog-8pr2c\" (UID: \"c55d597d-f42e-47c9-9825-3188bea3a6cb\") " pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" Jul 09 10:00:10.328089 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.328049 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" Jul 09 10:00:10.359306 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.359273 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8pr2c"] Jul 09 10:00:10.460395 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.460364 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8pr2c"] Jul 09 10:00:10.463555 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:10.463525 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55d597d_f42e_47c9_9825_3188bea3a6cb.slice/crio-a6f0198480240b35fb64f35459b0cfb662c6cd0eef51c4a2d26cb1455bae00a7 WatchSource:0}: Error finding container a6f0198480240b35fb64f35459b0cfb662c6cd0eef51c4a2d26cb1455bae00a7: Status 404 returned error can't find the container with id a6f0198480240b35fb64f35459b0cfb662c6cd0eef51c4a2d26cb1455bae00a7 Jul 09 10:00:10.570543 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.570513 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tqztm"] Jul 09 10:00:10.575007 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.574992 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:10.584407 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.584385 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tqztm"] Jul 09 10:00:10.726579 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.726540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxx9c\" (UniqueName: \"kubernetes.io/projected/0bbecb6e-e078-40c6-95c0-78b28144ce0d-kube-api-access-sxx9c\") pod \"kuadrant-operator-catalog-tqztm\" (UID: \"0bbecb6e-e078-40c6-95c0-78b28144ce0d\") " pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:10.782658 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.782622 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" event={"ID":"c55d597d-f42e-47c9-9825-3188bea3a6cb","Type":"ContainerStarted","Data":"a6f0198480240b35fb64f35459b0cfb662c6cd0eef51c4a2d26cb1455bae00a7"} Jul 09 10:00:10.827885 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.827852 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxx9c\" (UniqueName: \"kubernetes.io/projected/0bbecb6e-e078-40c6-95c0-78b28144ce0d-kube-api-access-sxx9c\") pod \"kuadrant-operator-catalog-tqztm\" (UID: \"0bbecb6e-e078-40c6-95c0-78b28144ce0d\") " pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:10.838896 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.838837 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxx9c\" (UniqueName: \"kubernetes.io/projected/0bbecb6e-e078-40c6-95c0-78b28144ce0d-kube-api-access-sxx9c\") pod \"kuadrant-operator-catalog-tqztm\" (UID: \"0bbecb6e-e078-40c6-95c0-78b28144ce0d\") " pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:10.884840 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:10.884797 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:11.020428 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:11.020401 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tqztm"] Jul 09 10:00:11.022249 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:11.022223 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbecb6e_e078_40c6_95c0_78b28144ce0d.slice/crio-2c387c96f7ad6282e2cfaafc920aaed08565f574fad891592c86159e0176dfb0 WatchSource:0}: Error finding container 2c387c96f7ad6282e2cfaafc920aaed08565f574fad891592c86159e0176dfb0: Status 404 returned error can't find the container with id 2c387c96f7ad6282e2cfaafc920aaed08565f574fad891592c86159e0176dfb0 Jul 09 10:00:11.790657 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:11.790617 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" event={"ID":"0bbecb6e-e078-40c6-95c0-78b28144ce0d","Type":"ContainerStarted","Data":"2c387c96f7ad6282e2cfaafc920aaed08565f574fad891592c86159e0176dfb0"} Jul 09 10:00:12.799314 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:12.799227 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" event={"ID":"c55d597d-f42e-47c9-9825-3188bea3a6cb","Type":"ContainerStarted","Data":"5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6"} Jul 09 10:00:12.799314 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:12.799288 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" podUID="c55d597d-f42e-47c9-9825-3188bea3a6cb" containerName="registry-server" containerID="cri-o://5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6" gracePeriod=2 Jul 09 10:00:12.800810 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:12.800782 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" event={"ID":"0bbecb6e-e078-40c6-95c0-78b28144ce0d","Type":"ContainerStarted","Data":"7d1eec5189d96f9d6f5a757cc7da50fc8ca188587dc9fda13510b67c0bee54e9"} Jul 09 10:00:12.819901 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:12.819858 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" podStartSLOduration=1.806142936 podStartE2EDuration="3.819845153s" totalImagesPullingTime="2.013702217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:00:12.817981801 +0000 UTC m=+368.150465599" watchObservedRunningTime="2026-07-09 10:00:12.819845153 +0000 UTC m=+368.152328969" Jul 09 10:00:12.836782 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:12.836733 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" podStartSLOduration=1.381081305 podStartE2EDuration="2.836720442s" totalImagesPullingTime="1.455639137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:00:12.83581158 +0000 UTC m=+368.168295378" watchObservedRunningTime="2026-07-09 10:00:12.836720442 +0000 UTC m=+368.169204240" Jul 09 10:00:13.043218 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.043193 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" Jul 09 10:00:13.148085 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.148046 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwbwv\" (UniqueName: \"kubernetes.io/projected/c55d597d-f42e-47c9-9825-3188bea3a6cb-kube-api-access-qwbwv\") pod \"c55d597d-f42e-47c9-9825-3188bea3a6cb\" (UID: \"c55d597d-f42e-47c9-9825-3188bea3a6cb\") " Jul 09 10:00:13.150247 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.150224 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55d597d-f42e-47c9-9825-3188bea3a6cb-kube-api-access-qwbwv" (OuterVolumeSpecName: "kube-api-access-qwbwv") pod "c55d597d-f42e-47c9-9825-3188bea3a6cb" (UID: "c55d597d-f42e-47c9-9825-3188bea3a6cb"). InnerVolumeSpecName "kube-api-access-qwbwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:00:13.249740 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.249699 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwbwv\" (UniqueName: \"kubernetes.io/projected/c55d597d-f42e-47c9-9825-3188bea3a6cb-kube-api-access-qwbwv\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:13.806325 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.806231 2571 generic.go:356] "Generic (PLEG): container finished" podID="c55d597d-f42e-47c9-9825-3188bea3a6cb" containerID="5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6" exitCode=0 Jul 09 10:00:13.806325 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.806298 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" Jul 09 10:00:13.806814 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.806320 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" event={"ID":"c55d597d-f42e-47c9-9825-3188bea3a6cb","Type":"ContainerDied","Data":"5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6"} Jul 09 10:00:13.806814 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.806404 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8pr2c" event={"ID":"c55d597d-f42e-47c9-9825-3188bea3a6cb","Type":"ContainerDied","Data":"a6f0198480240b35fb64f35459b0cfb662c6cd0eef51c4a2d26cb1455bae00a7"} Jul 09 10:00:13.806814 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.806434 2571 scope.go:117] "RemoveContainer" containerID="5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6" Jul 09 10:00:13.815813 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.815790 2571 scope.go:117] "RemoveContainer" containerID="5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6" Jul 09 10:00:13.816091 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:00:13.816073 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6\": container with ID starting with 5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6 not found: ID does not exist" containerID="5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6" Jul 09 10:00:13.816153 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.816103 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6"} err="failed to get container status \"5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6\": rpc error: code = NotFound desc = could not find container \"5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6\": container with ID starting with 5d098881aba77dd6853418df877868c79556daf4ead2817d78db56fe9752fda6 not found: ID does not exist" Jul 09 10:00:13.827003 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.826978 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8pr2c"] Jul 09 10:00:13.832020 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:13.831998 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8pr2c"] Jul 09 10:00:15.159777 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:15.159740 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c55d597d-f42e-47c9-9825-3188bea3a6cb" path="/var/lib/kubelet/pods/c55d597d-f42e-47c9-9825-3188bea3a6cb/volumes" Jul 09 10:00:20.885795 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:20.885756 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:20.886422 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:20.885825 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:20.908220 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:20.908195 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:21.870972 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:21.870942 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-tqztm" Jul 09 10:00:25.839828 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.839796 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp"] Jul 09 10:00:25.840211 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.840155 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c55d597d-f42e-47c9-9825-3188bea3a6cb" containerName="registry-server" Jul 09 10:00:25.840211 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.840170 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c55d597d-f42e-47c9-9825-3188bea3a6cb" containerName="registry-server" Jul 09 10:00:25.840292 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.840231 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c55d597d-f42e-47c9-9825-3188bea3a6cb" containerName="registry-server" Jul 09 10:00:25.844686 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.844668 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:25.847438 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.847417 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zvqhc\"" Jul 09 10:00:25.854718 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.854692 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp"] Jul 09 10:00:25.955229 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.955190 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:25.955423 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.955264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68l8g\" (UniqueName: \"kubernetes.io/projected/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-kube-api-access-68l8g\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:25.955423 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:25.955382 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:26.055770 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.055727 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68l8g\" (UniqueName: \"kubernetes.io/projected/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-kube-api-access-68l8g\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:26.055982 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.055792 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:26.055982 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.055816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:26.056210 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.056193 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:26.056276 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.056249 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:26.066568 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.066543 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68l8g\" (UniqueName: \"kubernetes.io/projected/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-kube-api-access-68l8g\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:26.154348 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.154253 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:26.302031 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.302001 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp"] Jul 09 10:00:26.304098 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:26.304067 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed4b1e8_5e49_4f14_9bf6_3e83d087c323.slice/crio-65687a350a9ae676917c95a508219e5f1e14ca4cf5cd21501a60f848654e5700 WatchSource:0}: Error finding container 65687a350a9ae676917c95a508219e5f1e14ca4cf5cd21501a60f848654e5700: Status 404 returned error can't find the container with id 65687a350a9ae676917c95a508219e5f1e14ca4cf5cd21501a60f848654e5700 Jul 09 10:00:26.424764 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.424731 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh"] Jul 09 10:00:26.428243 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.428224 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.442624 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.442593 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh"] Jul 09 10:00:26.459854 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.459823 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brv9h\" (UniqueName: \"kubernetes.io/projected/603117d7-d538-4e54-8f7f-ace4639e908d-kube-api-access-brv9h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.460012 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.459907 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.460012 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.459929 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.560537 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.560506 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brv9h\" (UniqueName: \"kubernetes.io/projected/603117d7-d538-4e54-8f7f-ace4639e908d-kube-api-access-brv9h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.560727 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.560581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.560727 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.560632 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.560954 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.560934 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.561015 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.560959 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.576991 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.576961 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brv9h\" (UniqueName: \"kubernetes.io/projected/603117d7-d538-4e54-8f7f-ace4639e908d-kube-api-access-brv9h\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.784738 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.784649 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:26.853279 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.853250 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5"] Jul 09 10:00:26.857977 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.857957 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.862044 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.862022 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.862187 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.862144 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.862187 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.862179 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xb6\" (UniqueName: \"kubernetes.io/projected/99254337-7c21-4570-a5c7-d2f3ea3598ec-kube-api-access-m8xb6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.872800 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.872764 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5"] Jul 09 10:00:26.877964 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.877936 2571 generic.go:356] "Generic (PLEG): container finished" podID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerID="b1460bdc5bec60f5009abeb94bf62a1b16409bf959fd6588e85dccdfeefacef6" exitCode=0 Jul 09 10:00:26.878086 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.878011 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" event={"ID":"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323","Type":"ContainerDied","Data":"b1460bdc5bec60f5009abeb94bf62a1b16409bf959fd6588e85dccdfeefacef6"} Jul 09 10:00:26.878086 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.878051 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" event={"ID":"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323","Type":"ContainerStarted","Data":"65687a350a9ae676917c95a508219e5f1e14ca4cf5cd21501a60f848654e5700"} Jul 09 10:00:26.934383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.934358 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh"] Jul 09 10:00:26.935977 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:26.935951 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603117d7_d538_4e54_8f7f_ace4639e908d.slice/crio-1b6bb11c3362afef048faff0bb5da9441e95eda4a2015a166e23dea27de29ce0 WatchSource:0}: Error finding container 1b6bb11c3362afef048faff0bb5da9441e95eda4a2015a166e23dea27de29ce0: Status 404 returned error can't find the container with id 1b6bb11c3362afef048faff0bb5da9441e95eda4a2015a166e23dea27de29ce0 Jul 09 10:00:26.962731 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.962697 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.962817 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.962737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xb6\" (UniqueName: \"kubernetes.io/projected/99254337-7c21-4570-a5c7-d2f3ea3598ec-kube-api-access-m8xb6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.962817 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.962759 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.963031 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.963014 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.963143 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.963103 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:26.981460 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:26.981435 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xb6\" (UniqueName: \"kubernetes.io/projected/99254337-7c21-4570-a5c7-d2f3ea3598ec-kube-api-access-m8xb6\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:27.166869 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.166840 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:27.254212 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.254177 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df"] Jul 09 10:00:27.259380 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.259355 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.265593 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.265563 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.265709 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.265613 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.265709 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.265687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pj4s\" (UniqueName: \"kubernetes.io/projected/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-kube-api-access-4pj4s\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.275521 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.275496 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df"] Jul 09 10:00:27.347007 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.346981 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5"] Jul 09 10:00:27.349368 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:27.349344 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99254337_7c21_4570_a5c7_d2f3ea3598ec.slice/crio-eeb6ac317baa097ceb93ba21bf48afe4a859bef8684f4a655723c1d88113889d WatchSource:0}: Error finding container eeb6ac317baa097ceb93ba21bf48afe4a859bef8684f4a655723c1d88113889d: Status 404 returned error can't find the container with id eeb6ac317baa097ceb93ba21bf48afe4a859bef8684f4a655723c1d88113889d Jul 09 10:00:27.366105 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.366080 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pj4s\" (UniqueName: \"kubernetes.io/projected/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-kube-api-access-4pj4s\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.366224 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.366203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.366291 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.366240 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.366556 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.366533 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.366624 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.366543 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.377051 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.377027 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pj4s\" (UniqueName: \"kubernetes.io/projected/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-kube-api-access-4pj4s\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.570939 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.570909 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:27.767011 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.766988 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df"] Jul 09 10:00:27.823951 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:27.823919 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c8a9fb_f58f_4093_bbbf_174e83759d0c.slice/crio-e2274a70e232c22493683574153cbc3b5102b25904753b95a885ccf5db7a6950 WatchSource:0}: Error finding container e2274a70e232c22493683574153cbc3b5102b25904753b95a885ccf5db7a6950: Status 404 returned error can't find the container with id e2274a70e232c22493683574153cbc3b5102b25904753b95a885ccf5db7a6950 Jul 09 10:00:27.883743 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.883667 2571 generic.go:356] "Generic (PLEG): container finished" podID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerID="f9469f624ea3e364e9881d2ce233fb682f4279e9a25a4d6683b9ce21ead4d8f3" exitCode=0 Jul 09 10:00:27.884185 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.883758 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" event={"ID":"99254337-7c21-4570-a5c7-d2f3ea3598ec","Type":"ContainerDied","Data":"f9469f624ea3e364e9881d2ce233fb682f4279e9a25a4d6683b9ce21ead4d8f3"} Jul 09 10:00:27.884185 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.883825 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" event={"ID":"99254337-7c21-4570-a5c7-d2f3ea3598ec","Type":"ContainerStarted","Data":"eeb6ac317baa097ceb93ba21bf48afe4a859bef8684f4a655723c1d88113889d"} Jul 09 10:00:27.885385 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.885359 2571 generic.go:356] "Generic (PLEG): container finished" podID="603117d7-d538-4e54-8f7f-ace4639e908d" containerID="a7ca7ee23d291be13244e3e50243a84b4b432d394ede9b66214b7ab007564657" exitCode=0 Jul 09 10:00:27.885487 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.885446 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" event={"ID":"603117d7-d538-4e54-8f7f-ace4639e908d","Type":"ContainerDied","Data":"a7ca7ee23d291be13244e3e50243a84b4b432d394ede9b66214b7ab007564657"} Jul 09 10:00:27.885487 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.885484 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" event={"ID":"603117d7-d538-4e54-8f7f-ace4639e908d","Type":"ContainerStarted","Data":"1b6bb11c3362afef048faff0bb5da9441e95eda4a2015a166e23dea27de29ce0"} Jul 09 10:00:27.887081 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.887057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" event={"ID":"c1c8a9fb-f58f-4093-bbbf-174e83759d0c","Type":"ContainerStarted","Data":"616574d46709d3933d85e835199a56f166bc05112e8ffe805ccaa650d50f52a9"} Jul 09 10:00:27.887215 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.887100 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" event={"ID":"c1c8a9fb-f58f-4093-bbbf-174e83759d0c","Type":"ContainerStarted","Data":"e2274a70e232c22493683574153cbc3b5102b25904753b95a885ccf5db7a6950"} Jul 09 10:00:27.888896 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.888876 2571 generic.go:356] "Generic (PLEG): container finished" podID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerID="272001ee149cf92c84b5ccc16d3e3f3055aac773068c52f02ae44e66829c2a64" exitCode=0 Jul 09 10:00:27.888993 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:27.888902 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" event={"ID":"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323","Type":"ContainerDied","Data":"272001ee149cf92c84b5ccc16d3e3f3055aac773068c52f02ae44e66829c2a64"} Jul 09 10:00:28.895075 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:28.895043 2571 generic.go:356] "Generic (PLEG): container finished" podID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerID="616574d46709d3933d85e835199a56f166bc05112e8ffe805ccaa650d50f52a9" exitCode=0 Jul 09 10:00:28.895533 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:28.895146 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" event={"ID":"c1c8a9fb-f58f-4093-bbbf-174e83759d0c","Type":"ContainerDied","Data":"616574d46709d3933d85e835199a56f166bc05112e8ffe805ccaa650d50f52a9"} Jul 09 10:00:28.897481 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:28.897421 2571 generic.go:356] "Generic (PLEG): container finished" podID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerID="70788331d6f9903c97d9f2451831281f5d9e8b8f2ce2d4f6ec39b77a9d5ae571" exitCode=0 Jul 09 10:00:28.897549 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:28.897485 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" event={"ID":"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323","Type":"ContainerDied","Data":"70788331d6f9903c97d9f2451831281f5d9e8b8f2ce2d4f6ec39b77a9d5ae571"} Jul 09 10:00:29.905852 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:29.905819 2571 generic.go:356] "Generic (PLEG): container finished" podID="603117d7-d538-4e54-8f7f-ace4639e908d" containerID="a2e73ec61eaf916d196be75a0cfb7939df685b25495310fb4f8bf61f3c6935b4" exitCode=0 Jul 09 10:00:29.906247 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:29.905899 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" event={"ID":"603117d7-d538-4e54-8f7f-ace4639e908d","Type":"ContainerDied","Data":"a2e73ec61eaf916d196be75a0cfb7939df685b25495310fb4f8bf61f3c6935b4"} Jul 09 10:00:30.049223 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.049194 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:30.084879 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.084853 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-util\") pod \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " Jul 09 10:00:30.085023 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.084889 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68l8g\" (UniqueName: \"kubernetes.io/projected/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-kube-api-access-68l8g\") pod \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " Jul 09 10:00:30.085023 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.084929 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-bundle\") pod \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\" (UID: \"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323\") " Jul 09 10:00:30.085491 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.085461 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-bundle" (OuterVolumeSpecName: "bundle") pod "8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" (UID: "8ed4b1e8-5e49-4f14-9bf6-3e83d087c323"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:00:30.087799 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.087759 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-kube-api-access-68l8g" (OuterVolumeSpecName: "kube-api-access-68l8g") pod "8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" (UID: "8ed4b1e8-5e49-4f14-9bf6-3e83d087c323"). InnerVolumeSpecName "kube-api-access-68l8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:00:30.090570 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.090542 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-util" (OuterVolumeSpecName: "util") pod "8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" (UID: "8ed4b1e8-5e49-4f14-9bf6-3e83d087c323"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:00:30.185750 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.185666 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-util\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:30.185750 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.185693 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68l8g\" (UniqueName: \"kubernetes.io/projected/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-kube-api-access-68l8g\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:30.185750 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.185705 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ed4b1e8-5e49-4f14-9bf6-3e83d087c323-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:30.913389 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.913356 2571 generic.go:356] "Generic (PLEG): container finished" podID="603117d7-d538-4e54-8f7f-ace4639e908d" containerID="fedf87f4b841001b9b707d6b003f37f24c6570b4dea36437f35c1755fb730a98" exitCode=0 Jul 09 10:00:30.913853 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.913482 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" event={"ID":"603117d7-d538-4e54-8f7f-ace4639e908d","Type":"ContainerDied","Data":"fedf87f4b841001b9b707d6b003f37f24c6570b4dea36437f35c1755fb730a98"} Jul 09 10:00:30.915336 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.915315 2571 generic.go:356] "Generic (PLEG): container finished" podID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerID="fb6e452b7f3631c1f95f8d0c0ad2f558b72d0562102830bc4027b5e53558a676" exitCode=0 Jul 09 10:00:30.915426 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.915398 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" event={"ID":"c1c8a9fb-f58f-4093-bbbf-174e83759d0c","Type":"ContainerDied","Data":"fb6e452b7f3631c1f95f8d0c0ad2f558b72d0562102830bc4027b5e53558a676"} Jul 09 10:00:30.917333 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.917313 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" event={"ID":"8ed4b1e8-5e49-4f14-9bf6-3e83d087c323","Type":"ContainerDied","Data":"65687a350a9ae676917c95a508219e5f1e14ca4cf5cd21501a60f848654e5700"} Jul 09 10:00:30.917406 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.917335 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp" Jul 09 10:00:30.917406 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:30.917350 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65687a350a9ae676917c95a508219e5f1e14ca4cf5cd21501a60f848654e5700" Jul 09 10:00:31.925293 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:31.925202 2571 generic.go:356] "Generic (PLEG): container finished" podID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerID="96262ceaf70e9328a237637595b376a997044475ccff1128b9a7230fa386123d" exitCode=0 Jul 09 10:00:31.925681 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:31.925292 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" event={"ID":"c1c8a9fb-f58f-4093-bbbf-174e83759d0c","Type":"ContainerDied","Data":"96262ceaf70e9328a237637595b376a997044475ccff1128b9a7230fa386123d"} Jul 09 10:00:31.926985 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:31.926962 2571 generic.go:356] "Generic (PLEG): container finished" podID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerID="3e1db60d0ed09bf47c1e694b830480e9a46c6d0cf2fd335453b1edbe4f36e7dc" exitCode=0 Jul 09 10:00:31.927057 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:31.927027 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" event={"ID":"99254337-7c21-4570-a5c7-d2f3ea3598ec","Type":"ContainerDied","Data":"3e1db60d0ed09bf47c1e694b830480e9a46c6d0cf2fd335453b1edbe4f36e7dc"} Jul 09 10:00:32.062836 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.062812 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:32.097658 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.097636 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brv9h\" (UniqueName: \"kubernetes.io/projected/603117d7-d538-4e54-8f7f-ace4639e908d-kube-api-access-brv9h\") pod \"603117d7-d538-4e54-8f7f-ace4639e908d\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " Jul 09 10:00:32.097773 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.097674 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-bundle\") pod \"603117d7-d538-4e54-8f7f-ace4639e908d\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " Jul 09 10:00:32.097773 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.097692 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-util\") pod \"603117d7-d538-4e54-8f7f-ace4639e908d\" (UID: \"603117d7-d538-4e54-8f7f-ace4639e908d\") " Jul 09 10:00:32.098306 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.098283 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-bundle" (OuterVolumeSpecName: "bundle") pod "603117d7-d538-4e54-8f7f-ace4639e908d" (UID: "603117d7-d538-4e54-8f7f-ace4639e908d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:00:32.103261 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.103236 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-util" (OuterVolumeSpecName: "util") pod "603117d7-d538-4e54-8f7f-ace4639e908d" (UID: "603117d7-d538-4e54-8f7f-ace4639e908d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:00:32.115476 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.115452 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603117d7-d538-4e54-8f7f-ace4639e908d-kube-api-access-brv9h" (OuterVolumeSpecName: "kube-api-access-brv9h") pod "603117d7-d538-4e54-8f7f-ace4639e908d" (UID: "603117d7-d538-4e54-8f7f-ace4639e908d"). InnerVolumeSpecName "kube-api-access-brv9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:00:32.198910 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.198813 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brv9h\" (UniqueName: \"kubernetes.io/projected/603117d7-d538-4e54-8f7f-ace4639e908d-kube-api-access-brv9h\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:32.198910 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.198851 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:32.198910 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.198866 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/603117d7-d538-4e54-8f7f-ace4639e908d-util\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:32.933951 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.933920 2571 generic.go:356] "Generic (PLEG): container finished" podID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerID="f1488195f2c175cf2269a75d3feb5565cac49bacdd817d1c1bffe5c31c596a9b" exitCode=0 Jul 09 10:00:32.934404 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.934001 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" event={"ID":"99254337-7c21-4570-a5c7-d2f3ea3598ec","Type":"ContainerDied","Data":"f1488195f2c175cf2269a75d3feb5565cac49bacdd817d1c1bffe5c31c596a9b"} Jul 09 10:00:32.935812 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.935782 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" event={"ID":"603117d7-d538-4e54-8f7f-ace4639e908d","Type":"ContainerDied","Data":"1b6bb11c3362afef048faff0bb5da9441e95eda4a2015a166e23dea27de29ce0"} Jul 09 10:00:32.935888 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.935811 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh" Jul 09 10:00:32.935888 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:32.935831 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b6bb11c3362afef048faff0bb5da9441e95eda4a2015a166e23dea27de29ce0" Jul 09 10:00:33.067905 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.067877 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:33.105899 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.105873 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-util\") pod \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " Jul 09 10:00:33.106074 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.105908 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pj4s\" (UniqueName: \"kubernetes.io/projected/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-kube-api-access-4pj4s\") pod \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " Jul 09 10:00:33.106074 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.106025 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-bundle\") pod \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\" (UID: \"c1c8a9fb-f58f-4093-bbbf-174e83759d0c\") " Jul 09 10:00:33.106573 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.106540 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-bundle" (OuterVolumeSpecName: "bundle") pod "c1c8a9fb-f58f-4093-bbbf-174e83759d0c" (UID: "c1c8a9fb-f58f-4093-bbbf-174e83759d0c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:00:33.108017 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.107990 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-kube-api-access-4pj4s" (OuterVolumeSpecName: "kube-api-access-4pj4s") pod "c1c8a9fb-f58f-4093-bbbf-174e83759d0c" (UID: "c1c8a9fb-f58f-4093-bbbf-174e83759d0c"). InnerVolumeSpecName "kube-api-access-4pj4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:00:33.111309 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.111287 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-util" (OuterVolumeSpecName: "util") pod "c1c8a9fb-f58f-4093-bbbf-174e83759d0c" (UID: "c1c8a9fb-f58f-4093-bbbf-174e83759d0c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:00:33.206779 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.206734 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-util\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:33.206779 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.206769 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pj4s\" (UniqueName: \"kubernetes.io/projected/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-kube-api-access-4pj4s\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:33.206779 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.206786 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1c8a9fb-f58f-4093-bbbf-174e83759d0c-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:33.942614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.942586 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" Jul 09 10:00:33.942614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.942601 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df" event={"ID":"c1c8a9fb-f58f-4093-bbbf-174e83759d0c","Type":"ContainerDied","Data":"e2274a70e232c22493683574153cbc3b5102b25904753b95a885ccf5db7a6950"} Jul 09 10:00:33.943059 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:33.942652 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2274a70e232c22493683574153cbc3b5102b25904753b95a885ccf5db7a6950" Jul 09 10:00:34.072838 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.072810 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:34.113941 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.113914 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-bundle\") pod \"99254337-7c21-4570-a5c7-d2f3ea3598ec\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " Jul 09 10:00:34.114082 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.113949 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-util\") pod \"99254337-7c21-4570-a5c7-d2f3ea3598ec\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " Jul 09 10:00:34.114082 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.113991 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xb6\" (UniqueName: \"kubernetes.io/projected/99254337-7c21-4570-a5c7-d2f3ea3598ec-kube-api-access-m8xb6\") pod \"99254337-7c21-4570-a5c7-d2f3ea3598ec\" (UID: \"99254337-7c21-4570-a5c7-d2f3ea3598ec\") " Jul 09 10:00:34.114498 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.114467 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-bundle" (OuterVolumeSpecName: "bundle") pod "99254337-7c21-4570-a5c7-d2f3ea3598ec" (UID: "99254337-7c21-4570-a5c7-d2f3ea3598ec"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:00:34.116164 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.116115 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99254337-7c21-4570-a5c7-d2f3ea3598ec-kube-api-access-m8xb6" (OuterVolumeSpecName: "kube-api-access-m8xb6") pod "99254337-7c21-4570-a5c7-d2f3ea3598ec" (UID: "99254337-7c21-4570-a5c7-d2f3ea3598ec"). InnerVolumeSpecName "kube-api-access-m8xb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:00:34.118819 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.118794 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-util" (OuterVolumeSpecName: "util") pod "99254337-7c21-4570-a5c7-d2f3ea3598ec" (UID: "99254337-7c21-4570-a5c7-d2f3ea3598ec"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:00:34.215311 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.215227 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:34.215311 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.215254 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/99254337-7c21-4570-a5c7-d2f3ea3598ec-util\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:34.215311 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.215263 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xb6\" (UniqueName: \"kubernetes.io/projected/99254337-7c21-4570-a5c7-d2f3ea3598ec-kube-api-access-m8xb6\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:00:34.949209 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.949176 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" Jul 09 10:00:34.949692 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.949180 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5" event={"ID":"99254337-7c21-4570-a5c7-d2f3ea3598ec","Type":"ContainerDied","Data":"eeb6ac317baa097ceb93ba21bf48afe4a859bef8684f4a655723c1d88113889d"} Jul 09 10:00:34.949692 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:34.949295 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeb6ac317baa097ceb93ba21bf48afe4a859bef8684f4a655723c1d88113889d" Jul 09 10:00:43.972990 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.972944 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6"] Jul 09 10:00:43.973634 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973610 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerName="pull" Jul 09 10:00:43.973706 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973638 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerName="pull" Jul 09 10:00:43.973706 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973668 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603117d7-d538-4e54-8f7f-ace4639e908d" containerName="util" Jul 09 10:00:43.973706 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973682 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="603117d7-d538-4e54-8f7f-ace4639e908d" containerName="util" Jul 09 10:00:43.973706 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973696 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603117d7-d538-4e54-8f7f-ace4639e908d" containerName="pull" Jul 09 10:00:43.973706 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973704 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="603117d7-d538-4e54-8f7f-ace4639e908d" containerName="pull" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973733 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerName="extract" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973742 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerName="extract" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973766 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="603117d7-d538-4e54-8f7f-ace4639e908d" containerName="extract" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973774 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="603117d7-d538-4e54-8f7f-ace4639e908d" containerName="extract" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973789 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerName="extract" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973805 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerName="extract" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973819 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerName="util" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973827 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerName="util" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973847 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerName="util" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973855 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerName="util" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973870 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerName="pull" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973878 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerName="pull" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973890 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerName="util" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973897 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerName="util" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973913 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerName="pull" Jul 09 10:00:43.973927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973921 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerName="pull" Jul 09 10:00:43.974614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973940 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerName="extract" Jul 09 10:00:43.974614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.973948 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerName="extract" Jul 09 10:00:43.974614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.974101 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="603117d7-d538-4e54-8f7f-ace4639e908d" containerName="extract" Jul 09 10:00:43.974614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.974113 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ed4b1e8-5e49-4f14-9bf6-3e83d087c323" containerName="extract" Jul 09 10:00:43.974614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.974138 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1c8a9fb-f58f-4093-bbbf-174e83759d0c" containerName="extract" Jul 09 10:00:43.974614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.974153 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="99254337-7c21-4570-a5c7-d2f3ea3598ec" containerName="extract" Jul 09 10:00:43.978286 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.978263 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:43.983536 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.983505 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zc2xd\"" Jul 09 10:00:43.996077 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.996054 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6"] Jul 09 10:00:43.997920 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.997892 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9ae897-787b-4748-a92e-77f1a33e531b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" (UID: \"9e9ae897-787b-4748-a92e-77f1a33e531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:43.998020 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:43.997932 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vmwg\" (UniqueName: \"kubernetes.io/projected/9e9ae897-787b-4748-a92e-77f1a33e531b-kube-api-access-6vmwg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" (UID: \"9e9ae897-787b-4748-a92e-77f1a33e531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:44.098496 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:44.098460 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9ae897-787b-4748-a92e-77f1a33e531b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" (UID: \"9e9ae897-787b-4748-a92e-77f1a33e531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:44.098496 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:44.098498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vmwg\" (UniqueName: \"kubernetes.io/projected/9e9ae897-787b-4748-a92e-77f1a33e531b-kube-api-access-6vmwg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" (UID: \"9e9ae897-787b-4748-a92e-77f1a33e531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:44.098833 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:44.098809 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9ae897-787b-4748-a92e-77f1a33e531b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" (UID: \"9e9ae897-787b-4748-a92e-77f1a33e531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:44.108812 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:44.108785 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vmwg\" (UniqueName: \"kubernetes.io/projected/9e9ae897-787b-4748-a92e-77f1a33e531b-kube-api-access-6vmwg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" (UID: \"9e9ae897-787b-4748-a92e-77f1a33e531b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:44.288095 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:44.288024 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:44.431046 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:44.431020 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6"] Jul 09 10:00:44.432869 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:44.432835 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9ae897_787b_4748_a92e_77f1a33e531b.slice/crio-b1b1720336fc6663fc3d473a505beb800a4a426e88062fd5f6957719aad811d6 WatchSource:0}: Error finding container b1b1720336fc6663fc3d473a505beb800a4a426e88062fd5f6957719aad811d6: Status 404 returned error can't find the container with id b1b1720336fc6663fc3d473a505beb800a4a426e88062fd5f6957719aad811d6 Jul 09 10:00:45.000778 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.000743 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" event={"ID":"9e9ae897-787b-4748-a92e-77f1a33e531b","Type":"ContainerStarted","Data":"b1b1720336fc6663fc3d473a505beb800a4a426e88062fd5f6957719aad811d6"} Jul 09 10:00:45.790403 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.790363 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n"] Jul 09 10:00:45.793885 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.793862 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" Jul 09 10:00:45.796648 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.796628 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 09 10:00:45.796729 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.796694 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9qnl6\"" Jul 09 10:00:45.809167 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.809145 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n"] Jul 09 10:00:45.811060 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.811039 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kl9d\" (UniqueName: \"kubernetes.io/projected/2bcae414-b097-45ff-8668-108bafb4a6b9-kube-api-access-9kl9d\") pod \"dns-operator-controller-manager-864dd78c4c-jx59n\" (UID: \"2bcae414-b097-45ff-8668-108bafb4a6b9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" Jul 09 10:00:45.911994 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.911963 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kl9d\" (UniqueName: \"kubernetes.io/projected/2bcae414-b097-45ff-8668-108bafb4a6b9-kube-api-access-9kl9d\") pod \"dns-operator-controller-manager-864dd78c4c-jx59n\" (UID: \"2bcae414-b097-45ff-8668-108bafb4a6b9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" Jul 09 10:00:45.924377 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:45.924343 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kl9d\" (UniqueName: \"kubernetes.io/projected/2bcae414-b097-45ff-8668-108bafb4a6b9-kube-api-access-9kl9d\") pod \"dns-operator-controller-manager-864dd78c4c-jx59n\" (UID: \"2bcae414-b097-45ff-8668-108bafb4a6b9\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" Jul 09 10:00:46.103901 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:46.103867 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" Jul 09 10:00:46.254340 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:46.254313 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n"] Jul 09 10:00:46.256318 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:46.256283 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcae414_b097_45ff_8668_108bafb4a6b9.slice/crio-259e5bd9436cfb115de485ca0305c5a31d6339b91aba2d703cd35eb969fdeae9 WatchSource:0}: Error finding container 259e5bd9436cfb115de485ca0305c5a31d6339b91aba2d703cd35eb969fdeae9: Status 404 returned error can't find the container with id 259e5bd9436cfb115de485ca0305c5a31d6339b91aba2d703cd35eb969fdeae9 Jul 09 10:00:47.011273 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:47.011229 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" event={"ID":"2bcae414-b097-45ff-8668-108bafb4a6b9","Type":"ContainerStarted","Data":"259e5bd9436cfb115de485ca0305c5a31d6339b91aba2d703cd35eb969fdeae9"} Jul 09 10:00:49.028551 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.028511 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" event={"ID":"2bcae414-b097-45ff-8668-108bafb4a6b9","Type":"ContainerStarted","Data":"3e0ee84efc0d86e55cddd5bd59a16b9b71a615c993fbb949e0f85f0fa2e85dc3"} Jul 09 10:00:49.028947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.028636 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" Jul 09 10:00:49.425119 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.425070 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" podStartSLOduration=2.463056088 podStartE2EDuration="4.425054142s" totalImagesPullingTime="1.961998054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:00:49.073800558 +0000 UTC m=+404.406284358" watchObservedRunningTime="2026-07-09 10:00:49.425054142 +0000 UTC m=+404.757537940" Jul 09 10:00:49.425586 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.425568 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vbfff"] Jul 09 10:00:49.429014 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.428995 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" Jul 09 10:00:49.435334 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.435312 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-r4qll\"" Jul 09 10:00:49.506075 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.506036 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66d476967d-lk75b"] Jul 09 10:00:49.509574 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.509536 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.531415 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.531390 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vbfff"] Jul 09 10:00:49.536079 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.536050 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d476967d-lk75b"] Jul 09 10:00:49.546061 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.546037 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e20594-dc24-4ddd-a6dc-980054710224-console-serving-cert\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.546236 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.546069 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-oauth-serving-cert\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.546236 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.546094 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-trusted-ca-bundle\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.546236 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.546177 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-console-config\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.546236 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.546205 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nkr7\" (UniqueName: \"kubernetes.io/projected/07e20594-dc24-4ddd-a6dc-980054710224-kube-api-access-9nkr7\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.546464 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.546252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854gp\" (UniqueName: \"kubernetes.io/projected/a11a241d-97f2-4ce8-af49-a556a3915b2f-kube-api-access-854gp\") pod \"authorino-operator-75cc8d5cb7-vbfff\" (UID: \"a11a241d-97f2-4ce8-af49-a556a3915b2f\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" Jul 09 10:00:49.546464 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.546289 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e20594-dc24-4ddd-a6dc-980054710224-console-oauth-config\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.546464 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.546328 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-service-ca\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.646943 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.646906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-service-ca\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.647142 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.646972 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e20594-dc24-4ddd-a6dc-980054710224-console-serving-cert\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.647142 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.647002 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-oauth-serving-cert\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.647142 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.647034 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-trusted-ca-bundle\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.647142 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.647069 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-console-config\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.647142 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.647091 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nkr7\" (UniqueName: \"kubernetes.io/projected/07e20594-dc24-4ddd-a6dc-980054710224-kube-api-access-9nkr7\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.647142 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.647139 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854gp\" (UniqueName: \"kubernetes.io/projected/a11a241d-97f2-4ce8-af49-a556a3915b2f-kube-api-access-854gp\") pod \"authorino-operator-75cc8d5cb7-vbfff\" (UID: \"a11a241d-97f2-4ce8-af49-a556a3915b2f\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" Jul 09 10:00:49.647471 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.647186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e20594-dc24-4ddd-a6dc-980054710224-console-oauth-config\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.647854 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.647808 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-service-ca\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.647854 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.647831 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-oauth-serving-cert\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.648280 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.648232 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-console-config\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.648516 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.648493 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07e20594-dc24-4ddd-a6dc-980054710224-trusted-ca-bundle\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.650156 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.650105 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07e20594-dc24-4ddd-a6dc-980054710224-console-serving-cert\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.650311 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.650290 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07e20594-dc24-4ddd-a6dc-980054710224-console-oauth-config\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.665041 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.665016 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nkr7\" (UniqueName: \"kubernetes.io/projected/07e20594-dc24-4ddd-a6dc-980054710224-kube-api-access-9nkr7\") pod \"console-66d476967d-lk75b\" (UID: \"07e20594-dc24-4ddd-a6dc-980054710224\") " pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:49.666362 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.666341 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854gp\" (UniqueName: \"kubernetes.io/projected/a11a241d-97f2-4ce8-af49-a556a3915b2f-kube-api-access-854gp\") pod \"authorino-operator-75cc8d5cb7-vbfff\" (UID: \"a11a241d-97f2-4ce8-af49-a556a3915b2f\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" Jul 09 10:00:49.739355 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.739244 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" Jul 09 10:00:49.821329 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:49.821297 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:50.921315 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:50.921292 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vbfff"] Jul 09 10:00:50.923869 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:50.923845 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11a241d_97f2_4ce8_af49_a556a3915b2f.slice/crio-b3227f2f61af4c33cd4a4fdd8ad04615c3298674515c2087bebc903d1370a9bb WatchSource:0}: Error finding container b3227f2f61af4c33cd4a4fdd8ad04615c3298674515c2087bebc903d1370a9bb: Status 404 returned error can't find the container with id b3227f2f61af4c33cd4a4fdd8ad04615c3298674515c2087bebc903d1370a9bb Jul 09 10:00:50.944353 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:50.944299 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d476967d-lk75b"] Jul 09 10:00:50.946522 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:50.946495 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e20594_dc24_4ddd_a6dc_980054710224.slice/crio-fc2766d8c2d75bb89d171cb963a417a63890a303fe85b5731aaffcd1927c4256 WatchSource:0}: Error finding container fc2766d8c2d75bb89d171cb963a417a63890a303fe85b5731aaffcd1927c4256: Status 404 returned error can't find the container with id fc2766d8c2d75bb89d171cb963a417a63890a303fe85b5731aaffcd1927c4256 Jul 09 10:00:51.043367 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:51.043284 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d476967d-lk75b" event={"ID":"07e20594-dc24-4ddd-a6dc-980054710224","Type":"ContainerStarted","Data":"ed3d8edfa7b4dbfdea123d47f00ae3bf8e725286a943f492401c184d8b6a00c0"} Jul 09 10:00:51.043367 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:51.043340 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d476967d-lk75b" event={"ID":"07e20594-dc24-4ddd-a6dc-980054710224","Type":"ContainerStarted","Data":"fc2766d8c2d75bb89d171cb963a417a63890a303fe85b5731aaffcd1927c4256"} Jul 09 10:00:51.045015 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:51.044985 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" event={"ID":"a11a241d-97f2-4ce8-af49-a556a3915b2f","Type":"ContainerStarted","Data":"b3227f2f61af4c33cd4a4fdd8ad04615c3298674515c2087bebc903d1370a9bb"} Jul 09 10:00:51.046905 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:51.046880 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" event={"ID":"9e9ae897-787b-4748-a92e-77f1a33e531b","Type":"ContainerStarted","Data":"b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92"} Jul 09 10:00:51.047032 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:51.047016 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:00:51.071146 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:51.071067 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66d476967d-lk75b" podStartSLOduration=2.071051731 podStartE2EDuration="2.071051731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:00:51.069529083 +0000 UTC m=+406.402012885" watchObservedRunningTime="2026-07-09 10:00:51.071051731 +0000 UTC m=+406.403535510" Jul 09 10:00:51.098686 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:51.098635 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" podStartSLOduration=1.703682331 podStartE2EDuration="8.098619049s" totalImagesPullingTime="6.394936718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:00:51.096677255 +0000 UTC m=+406.429161052" watchObservedRunningTime="2026-07-09 10:00:51.098619049 +0000 UTC m=+406.431102914" Jul 09 10:00:52.244590 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.244453 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68"] Jul 09 10:00:52.248456 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.248434 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:00:52.252938 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.252913 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bjgfm\"" Jul 09 10:00:52.270419 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.270394 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68"] Jul 09 10:00:52.375011 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.374968 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hp6v\" (UniqueName: \"kubernetes.io/projected/01fb9185-a31e-4733-b21f-01bd6858c020-kube-api-access-6hp6v\") pod \"limitador-operator-controller-manager-859d7b78bd-ldt68\" (UID: \"01fb9185-a31e-4733-b21f-01bd6858c020\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:00:52.475883 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.475829 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hp6v\" (UniqueName: \"kubernetes.io/projected/01fb9185-a31e-4733-b21f-01bd6858c020-kube-api-access-6hp6v\") pod \"limitador-operator-controller-manager-859d7b78bd-ldt68\" (UID: \"01fb9185-a31e-4733-b21f-01bd6858c020\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:00:52.495498 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.495428 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hp6v\" (UniqueName: \"kubernetes.io/projected/01fb9185-a31e-4733-b21f-01bd6858c020-kube-api-access-6hp6v\") pod \"limitador-operator-controller-manager-859d7b78bd-ldt68\" (UID: \"01fb9185-a31e-4733-b21f-01bd6858c020\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:00:52.563465 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.563430 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:00:52.732871 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:52.732836 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68"] Jul 09 10:00:52.734823 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:00:52.734760 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fb9185_a31e_4733_b21f_01bd6858c020.slice/crio-aee734959c096878b7e20679b64bbb45fa181ddb504dd7e4315e250872bc230f WatchSource:0}: Error finding container aee734959c096878b7e20679b64bbb45fa181ddb504dd7e4315e250872bc230f: Status 404 returned error can't find the container with id aee734959c096878b7e20679b64bbb45fa181ddb504dd7e4315e250872bc230f Jul 09 10:00:53.066733 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:53.066696 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" event={"ID":"01fb9185-a31e-4733-b21f-01bd6858c020","Type":"ContainerStarted","Data":"aee734959c096878b7e20679b64bbb45fa181ddb504dd7e4315e250872bc230f"} Jul 09 10:00:55.081947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:55.081899 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" event={"ID":"a11a241d-97f2-4ce8-af49-a556a3915b2f","Type":"ContainerStarted","Data":"1eda2b1348e8ebd13def997bc1bd2cfe5aa74e7648e937b10838aab7b888a25d"} Jul 09 10:00:55.082429 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:55.081970 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" Jul 09 10:00:56.088324 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:56.088276 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" event={"ID":"01fb9185-a31e-4733-b21f-01bd6858c020","Type":"ContainerStarted","Data":"e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb"} Jul 09 10:00:56.088736 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:56.088489 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:00:56.124425 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:56.124378 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" podStartSLOduration=4.188415478 podStartE2EDuration="7.124362674s" totalImagesPullingTime="2.935947196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:00:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:00:55.13186077 +0000 UTC m=+410.464344567" watchObservedRunningTime="2026-07-09 10:00:56.124362674 +0000 UTC m=+411.456846471" Jul 09 10:00:59.822255 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:59.822223 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:59.822611 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:59.822306 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:59.827385 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:59.827361 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:00:59.869082 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:00:59.869032 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" podStartSLOduration=5.427454434 podStartE2EDuration="7.869017332s" totalImagesPullingTime="2.441562898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:00:56.128218117 +0000 UTC m=+411.460701928" watchObservedRunningTime="2026-07-09 10:00:59.869017332 +0000 UTC m=+415.201501133" Jul 09 10:01:00.036794 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:00.036760 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-jx59n" Jul 09 10:01:00.119854 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:00.119765 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66d476967d-lk75b" Jul 09 10:01:00.177443 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:00.177406 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d85fbc597-gpxsd"] Jul 09 10:01:02.060947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:02.060914 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:03.823777 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.823739 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6"] Jul 09 10:01:03.824192 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.823936 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" containerName="manager" containerID="cri-o://b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92" gracePeriod=2 Jul 09 10:01:03.837183 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.837105 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6"] Jul 09 10:01:03.848696 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.848669 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68"] Jul 09 10:01:03.848926 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.848903 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" containerName="manager" containerID="cri-o://e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb" gracePeriod=2 Jul 09 10:01:03.851263 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.851238 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:03.852198 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:03.852091 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:03.852895 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.852843 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68"] Jul 09 10:01:03.861289 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:03.861260 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:03.863684 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:03.863659 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:03.867907 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.867886 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm"] Jul 09 10:01:03.868245 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.868232 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" containerName="manager" Jul 09 10:01:03.868291 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.868247 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" containerName="manager" Jul 09 10:01:03.868291 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.868260 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" containerName="manager" Jul 09 10:01:03.868291 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.868265 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" containerName="manager" Jul 09 10:01:03.868383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.868318 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" containerName="manager" Jul 09 10:01:03.868383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.868326 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" containerName="manager" Jul 09 10:01:03.871366 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.871349 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:03.889509 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:03.889470 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:03.893417 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:03.893393 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:03.895786 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:03.895754 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:03.907712 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.907689 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv"] Jul 09 10:01:03.911167 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.911147 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" Jul 09 10:01:03.911270 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.911181 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm"] Jul 09 10:01:03.921823 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.921796 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv"] Jul 09 10:01:03.960140 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:03.960095 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:03.977798 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.977770 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d8503b-0b68-4f75-a9fe-117c651bdab9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlkpm\" (UID: \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:03.977947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.977821 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxwn\" (UniqueName: \"kubernetes.io/projected/719c52a8-0b0a-44c8-a150-bb5b4d5422e8-kube-api-access-6cxwn\") pod \"limitador-operator-controller-manager-859d7b78bd-4g2mv\" (UID: \"719c52a8-0b0a-44c8-a150-bb5b4d5422e8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" Jul 09 10:01:03.977947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:03.977848 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28s2\" (UniqueName: \"kubernetes.io/projected/c5d8503b-0b68-4f75-a9fe-117c651bdab9-kube-api-access-q28s2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlkpm\" (UID: \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:03.999052 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:03.999016 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:04.078578 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.078529 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d8503b-0b68-4f75-a9fe-117c651bdab9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlkpm\" (UID: \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:04.078736 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.078616 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxwn\" (UniqueName: \"kubernetes.io/projected/719c52a8-0b0a-44c8-a150-bb5b4d5422e8-kube-api-access-6cxwn\") pod \"limitador-operator-controller-manager-859d7b78bd-4g2mv\" (UID: \"719c52a8-0b0a-44c8-a150-bb5b4d5422e8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" Jul 09 10:01:04.078736 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.078663 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28s2\" (UniqueName: \"kubernetes.io/projected/c5d8503b-0b68-4f75-a9fe-117c651bdab9-kube-api-access-q28s2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlkpm\" (UID: \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:04.078948 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.078927 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d8503b-0b68-4f75-a9fe-117c651bdab9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlkpm\" (UID: \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:04.090081 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.090059 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxwn\" (UniqueName: \"kubernetes.io/projected/719c52a8-0b0a-44c8-a150-bb5b4d5422e8-kube-api-access-6cxwn\") pod \"limitador-operator-controller-manager-859d7b78bd-4g2mv\" (UID: \"719c52a8-0b0a-44c8-a150-bb5b4d5422e8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" Jul 09 10:01:04.090188 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.090153 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28s2\" (UniqueName: \"kubernetes.io/projected/c5d8503b-0b68-4f75-a9fe-117c651bdab9-kube-api-access-q28s2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nlkpm\" (UID: \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:04.107018 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.106998 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:04.109151 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.109113 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:04.109359 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.109339 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:04.111690 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.111670 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:04.113789 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.113764 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:04.115779 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.115755 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:04.138551 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.138527 2571 generic.go:356] "Generic (PLEG): container finished" podID="9e9ae897-787b-4748-a92e-77f1a33e531b" containerID="b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92" exitCode=0 Jul 09 10:01:04.138642 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.138564 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:04.138642 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.138621 2571 scope.go:117] "RemoveContainer" containerID="b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92" Jul 09 10:01:04.140058 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.140035 2571 generic.go:356] "Generic (PLEG): container finished" podID="01fb9185-a31e-4733-b21f-01bd6858c020" containerID="e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb" exitCode=0 Jul 09 10:01:04.140152 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.140071 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:04.140920 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.140899 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:04.142842 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.142815 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:04.144946 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.144926 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:04.147192 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.147158 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:04.147268 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.147253 2571 scope.go:117] "RemoveContainer" containerID="b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92" Jul 09 10:01:04.147502 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.147482 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92\": container with ID starting with b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92 not found: ID does not exist" containerID="b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92" Jul 09 10:01:04.147547 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.147511 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92"} err="failed to get container status \"b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92\": rpc error: code = NotFound desc = could not find container \"b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92\": container with ID starting with b1ca731e855a53fcb574d46f4f557d28ee3fcb0f77c200101dbd151c3f8b3a92 not found: ID does not exist" Jul 09 10:01:04.147547 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.147527 2571 scope.go:117] "RemoveContainer" containerID="e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb" Jul 09 10:01:04.155302 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.155266 2571 scope.go:117] "RemoveContainer" containerID="e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb" Jul 09 10:01:04.155541 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.155522 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb\": container with ID starting with e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb not found: ID does not exist" containerID="e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb" Jul 09 10:01:04.155616 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.155545 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb"} err="failed to get container status \"e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb\": rpc error: code = NotFound desc = could not find container \"e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb\": container with ID starting with e1ca4d190f070a1a40fa4a049e39084342cc9a438b8172fe5c89b251e5e879bb not found: ID does not exist" Jul 09 10:01:04.179963 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.179942 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9ae897-787b-4748-a92e-77f1a33e531b-extensions-socket-volume\") pod \"9e9ae897-787b-4748-a92e-77f1a33e531b\" (UID: \"9e9ae897-787b-4748-a92e-77f1a33e531b\") " Jul 09 10:01:04.180074 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.179989 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vmwg\" (UniqueName: \"kubernetes.io/projected/9e9ae897-787b-4748-a92e-77f1a33e531b-kube-api-access-6vmwg\") pod \"9e9ae897-787b-4748-a92e-77f1a33e531b\" (UID: \"9e9ae897-787b-4748-a92e-77f1a33e531b\") " Jul 09 10:01:04.180074 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.180061 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hp6v\" (UniqueName: \"kubernetes.io/projected/01fb9185-a31e-4733-b21f-01bd6858c020-kube-api-access-6hp6v\") pod \"01fb9185-a31e-4733-b21f-01bd6858c020\" (UID: \"01fb9185-a31e-4733-b21f-01bd6858c020\") " Jul 09 10:01:04.180451 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.180429 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e9ae897-787b-4748-a92e-77f1a33e531b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9e9ae897-787b-4748-a92e-77f1a33e531b" (UID: "9e9ae897-787b-4748-a92e-77f1a33e531b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:01:04.181925 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.181906 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9ae897-787b-4748-a92e-77f1a33e531b-kube-api-access-6vmwg" (OuterVolumeSpecName: "kube-api-access-6vmwg") pod "9e9ae897-787b-4748-a92e-77f1a33e531b" (UID: "9e9ae897-787b-4748-a92e-77f1a33e531b"). InnerVolumeSpecName "kube-api-access-6vmwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:01:04.182000 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.181950 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fb9185-a31e-4733-b21f-01bd6858c020-kube-api-access-6hp6v" (OuterVolumeSpecName: "kube-api-access-6hp6v") pod "01fb9185-a31e-4733-b21f-01bd6858c020" (UID: "01fb9185-a31e-4733-b21f-01bd6858c020"). InnerVolumeSpecName "kube-api-access-6hp6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:01:04.270280 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.270246 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:04.276547 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.276526 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" Jul 09 10:01:04.280647 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.280622 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hp6v\" (UniqueName: \"kubernetes.io/projected/01fb9185-a31e-4733-b21f-01bd6858c020-kube-api-access-6hp6v\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:04.280780 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.280650 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e9ae897-787b-4748-a92e-77f1a33e531b-extensions-socket-volume\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:04.280780 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.280664 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vmwg\" (UniqueName: \"kubernetes.io/projected/9e9ae897-787b-4748-a92e-77f1a33e531b-kube-api-access-6vmwg\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:04.447752 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.447727 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm"] Jul 09 10:01:04.450849 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:01:04.450808 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d8503b_0b68_4f75_a9fe_117c651bdab9.slice/crio-7d38c367261ee2c6aa59ac1b28d9a566c8fe4d027f59c49f8d150a6276d9f461 WatchSource:0}: Error finding container 7d38c367261ee2c6aa59ac1b28d9a566c8fe4d027f59c49f8d150a6276d9f461: Status 404 returned error can't find the container with id 7d38c367261ee2c6aa59ac1b28d9a566c8fe4d027f59c49f8d150a6276d9f461 Jul 09 10:01:04.453175 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:04.453150 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv"] Jul 09 10:01:04.454446 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:01:04.454416 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719c52a8_0b0a_44c8_a150_bb5b4d5422e8.slice/crio-48e689375079beab7ea1aba7a27eddaaeb84e357b33b0ccf8bc14fe0e5770073 WatchSource:0}: Error finding container 48e689375079beab7ea1aba7a27eddaaeb84e357b33b0ccf8bc14fe0e5770073: Status 404 returned error can't find the container with id 48e689375079beab7ea1aba7a27eddaaeb84e357b33b0ccf8bc14fe0e5770073 Jul 09 10:01:04.455025 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.454988 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:04.457434 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.457405 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:04.459387 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.459362 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:04.461344 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:04.461278 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:05.147180 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.147137 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" event={"ID":"c5d8503b-0b68-4f75-a9fe-117c651bdab9","Type":"ContainerStarted","Data":"b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417"} Jul 09 10:01:05.147604 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.147243 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:05.147604 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.147268 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" event={"ID":"c5d8503b-0b68-4f75-a9fe-117c651bdab9","Type":"ContainerStarted","Data":"7d38c367261ee2c6aa59ac1b28d9a566c8fe4d027f59c49f8d150a6276d9f461"} Jul 09 10:01:05.148505 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.148479 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" event={"ID":"719c52a8-0b0a-44c8-a150-bb5b4d5422e8","Type":"ContainerStarted","Data":"01cea6ce8a355b5721c751152e59afeea7ec19334967a0f971dce6cc740d8376"} Jul 09 10:01:05.148593 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.148519 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" event={"ID":"719c52a8-0b0a-44c8-a150-bb5b4d5422e8","Type":"ContainerStarted","Data":"48e689375079beab7ea1aba7a27eddaaeb84e357b33b0ccf8bc14fe0e5770073"} Jul 09 10:01:05.148646 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.148619 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" Jul 09 10:01:05.149736 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:05.149704 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-ldt68\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-ldt68" Jul 09 10:01:05.161465 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.161440 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01fb9185-a31e-4733-b21f-01bd6858c020" path="/var/lib/kubelet/pods/01fb9185-a31e-4733-b21f-01bd6858c020/volumes" Jul 09 10:01:05.161851 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.161833 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" path="/var/lib/kubelet/pods/9e9ae897-787b-4748-a92e-77f1a33e531b/volumes" Jul 09 10:01:05.177856 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:05.177827 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-v4lv6\" is forbidden: User \"system:node:ip-10-0-140-69.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-69.ec2.internal' and this object" podUID="9e9ae897-787b-4748-a92e-77f1a33e531b" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-v4lv6" Jul 09 10:01:05.178091 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.178048 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" podStartSLOduration=2.178035807 podStartE2EDuration="2.178035807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:01:05.175457033 +0000 UTC m=+420.507940838" watchObservedRunningTime="2026-07-09 10:01:05.178035807 +0000 UTC m=+420.510519615" Jul 09 10:01:05.208260 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.208146 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" podStartSLOduration=2.208106622 podStartE2EDuration="2.208106622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:01:05.207417688 +0000 UTC m=+420.539901488" watchObservedRunningTime="2026-07-09 10:01:05.208106622 +0000 UTC m=+420.540590422" Jul 09 10:01:05.494590 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.494508 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt"] Jul 09 10:01:05.499461 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.499439 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:05.514535 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.514511 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt"] Jul 09 10:01:05.596757 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.596727 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzms\" (UniqueName: \"kubernetes.io/projected/c33dcc2b-9100-48cd-a018-f1aff926cce6-kube-api-access-jkzms\") pod \"kuadrant-operator-controller-manager-86878c769f-b8pxt\" (UID: \"c33dcc2b-9100-48cd-a018-f1aff926cce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:05.596920 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.596768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33dcc2b-9100-48cd-a018-f1aff926cce6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b8pxt\" (UID: \"c33dcc2b-9100-48cd-a018-f1aff926cce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:05.697680 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.697638 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33dcc2b-9100-48cd-a018-f1aff926cce6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b8pxt\" (UID: \"c33dcc2b-9100-48cd-a018-f1aff926cce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:05.697852 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.697746 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzms\" (UniqueName: \"kubernetes.io/projected/c33dcc2b-9100-48cd-a018-f1aff926cce6-kube-api-access-jkzms\") pod \"kuadrant-operator-controller-manager-86878c769f-b8pxt\" (UID: \"c33dcc2b-9100-48cd-a018-f1aff926cce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:05.698067 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.698047 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33dcc2b-9100-48cd-a018-f1aff926cce6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b8pxt\" (UID: \"c33dcc2b-9100-48cd-a018-f1aff926cce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:05.709582 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.709550 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzms\" (UniqueName: \"kubernetes.io/projected/c33dcc2b-9100-48cd-a018-f1aff926cce6-kube-api-access-jkzms\") pod \"kuadrant-operator-controller-manager-86878c769f-b8pxt\" (UID: \"c33dcc2b-9100-48cd-a018-f1aff926cce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:05.809775 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.809690 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:05.946807 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:05.946781 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt"] Jul 09 10:01:05.948959 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:01:05.948933 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33dcc2b_9100_48cd_a018_f1aff926cce6.slice/crio-05a8fb6f060ad2ce48c48136544c4ee7208c63ed657dff394be1b7ee77e97fac WatchSource:0}: Error finding container 05a8fb6f060ad2ce48c48136544c4ee7208c63ed657dff394be1b7ee77e97fac: Status 404 returned error can't find the container with id 05a8fb6f060ad2ce48c48136544c4ee7208c63ed657dff394be1b7ee77e97fac Jul 09 10:01:06.090673 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:06.090644 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbfff" Jul 09 10:01:06.156631 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:06.156584 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" event={"ID":"c33dcc2b-9100-48cd-a018-f1aff926cce6","Type":"ContainerStarted","Data":"a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231"} Jul 09 10:01:06.157003 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:06.156656 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" event={"ID":"c33dcc2b-9100-48cd-a018-f1aff926cce6","Type":"ContainerStarted","Data":"05a8fb6f060ad2ce48c48136544c4ee7208c63ed657dff394be1b7ee77e97fac"} Jul 09 10:01:06.185445 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:06.185395 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" podStartSLOduration=1.185377718 podStartE2EDuration="1.185377718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:01:06.183762411 +0000 UTC m=+421.516246209" watchObservedRunningTime="2026-07-09 10:01:06.185377718 +0000 UTC m=+421.517861517" Jul 09 10:01:07.162727 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:07.162696 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:16.159454 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:16.159421 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:16.159819 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:16.159489 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4g2mv" Jul 09 10:01:18.170794 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.170759 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:01:18.231231 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.231200 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm"] Jul 09 10:01:18.231460 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.231420 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" podUID="c5d8503b-0b68-4f75-a9fe-117c651bdab9" containerName="manager" containerID="cri-o://b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417" gracePeriod=10 Jul 09 10:01:18.487355 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.487327 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:18.510502 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.510475 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q28s2\" (UniqueName: \"kubernetes.io/projected/c5d8503b-0b68-4f75-a9fe-117c651bdab9-kube-api-access-q28s2\") pod \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\" (UID: \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\") " Jul 09 10:01:18.510652 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.510572 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d8503b-0b68-4f75-a9fe-117c651bdab9-extensions-socket-volume\") pod \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\" (UID: \"c5d8503b-0b68-4f75-a9fe-117c651bdab9\") " Jul 09 10:01:18.510906 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.510884 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d8503b-0b68-4f75-a9fe-117c651bdab9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c5d8503b-0b68-4f75-a9fe-117c651bdab9" (UID: "c5d8503b-0b68-4f75-a9fe-117c651bdab9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:01:18.512513 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.512489 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d8503b-0b68-4f75-a9fe-117c651bdab9-kube-api-access-q28s2" (OuterVolumeSpecName: "kube-api-access-q28s2") pod "c5d8503b-0b68-4f75-a9fe-117c651bdab9" (UID: "c5d8503b-0b68-4f75-a9fe-117c651bdab9"). InnerVolumeSpecName "kube-api-access-q28s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:01:18.611812 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.611779 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c5d8503b-0b68-4f75-a9fe-117c651bdab9-extensions-socket-volume\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:18.611812 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:18.611804 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q28s2\" (UniqueName: \"kubernetes.io/projected/c5d8503b-0b68-4f75-a9fe-117c651bdab9-kube-api-access-q28s2\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:19.223249 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.223210 2571 generic.go:356] "Generic (PLEG): container finished" podID="c5d8503b-0b68-4f75-a9fe-117c651bdab9" containerID="b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417" exitCode=0 Jul 09 10:01:19.223692 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.223253 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" Jul 09 10:01:19.223692 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.223254 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" event={"ID":"c5d8503b-0b68-4f75-a9fe-117c651bdab9","Type":"ContainerDied","Data":"b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417"} Jul 09 10:01:19.223692 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.223318 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm" event={"ID":"c5d8503b-0b68-4f75-a9fe-117c651bdab9","Type":"ContainerDied","Data":"7d38c367261ee2c6aa59ac1b28d9a566c8fe4d027f59c49f8d150a6276d9f461"} Jul 09 10:01:19.223692 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.223353 2571 scope.go:117] "RemoveContainer" containerID="b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417" Jul 09 10:01:19.232064 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.231857 2571 scope.go:117] "RemoveContainer" containerID="b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417" Jul 09 10:01:19.232163 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:19.232146 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417\": container with ID starting with b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417 not found: ID does not exist" containerID="b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417" Jul 09 10:01:19.232211 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.232173 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417"} err="failed to get container status \"b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417\": rpc error: code = NotFound desc = could not find container \"b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417\": container with ID starting with b3c3ff880a0ad88e0d9bca7fb373de82b012f9ed541bc940bfe57e29511c4417 not found: ID does not exist" Jul 09 10:01:19.246035 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.246007 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm"] Jul 09 10:01:19.252507 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:19.252486 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nlkpm"] Jul 09 10:01:21.160355 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:21.160322 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5d8503b-0b68-4f75-a9fe-117c651bdab9" path="/var/lib/kubelet/pods/c5d8503b-0b68-4f75-a9fe-117c651bdab9/volumes" Jul 09 10:01:25.209430 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.209390 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6d85fbc597-gpxsd" podUID="7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" containerName="console" containerID="cri-o://e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0" gracePeriod=15 Jul 09 10:01:25.493342 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.493316 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 10:01:25.559066 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559026 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-service-ca\") pod \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " Jul 09 10:01:25.559280 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559081 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-config\") pod \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " Jul 09 10:01:25.559280 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559113 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzzkq\" (UniqueName: \"kubernetes.io/projected/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-kube-api-access-bzzkq\") pod \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " Jul 09 10:01:25.559280 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559171 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-oauth-config\") pod \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " Jul 09 10:01:25.559280 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559196 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-serving-cert\") pod \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " Jul 09 10:01:25.559280 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559240 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-trusted-ca-bundle\") pod \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " Jul 09 10:01:25.559280 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559271 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-oauth-serving-cert\") pod \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\" (UID: \"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499\") " Jul 09 10:01:25.559583 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559498 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-config" (OuterVolumeSpecName: "console-config") pod "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" (UID: "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:01:25.559583 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559547 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-service-ca" (OuterVolumeSpecName: "service-ca") pod "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" (UID: "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:01:25.559801 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559773 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" (UID: "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:01:25.559880 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.559798 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" (UID: "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:01:25.561427 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.561397 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-kube-api-access-bzzkq" (OuterVolumeSpecName: "kube-api-access-bzzkq") pod "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" (UID: "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499"). InnerVolumeSpecName "kube-api-access-bzzkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:01:25.561530 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.561471 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" (UID: "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:01:25.561530 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.561488 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" (UID: "7a0c84ed-49e0-470a-8ccc-a08f2cd5a499"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:01:25.660715 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.660681 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-trusted-ca-bundle\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:25.660715 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.660709 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-oauth-serving-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:25.660715 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.660719 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:25.660940 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.660728 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-config\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:25.660940 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.660736 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzzkq\" (UniqueName: \"kubernetes.io/projected/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-kube-api-access-bzzkq\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:25.660940 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.660768 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-oauth-config\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:25.660940 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:25.660779 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499-console-serving-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:26.262414 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.262376 2571 generic.go:356] "Generic (PLEG): container finished" podID="7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" containerID="e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0" exitCode=0 Jul 09 10:01:26.262820 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.262442 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d85fbc597-gpxsd" Jul 09 10:01:26.262820 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.262464 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d85fbc597-gpxsd" event={"ID":"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499","Type":"ContainerDied","Data":"e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0"} Jul 09 10:01:26.262820 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.262518 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d85fbc597-gpxsd" event={"ID":"7a0c84ed-49e0-470a-8ccc-a08f2cd5a499","Type":"ContainerDied","Data":"a8283824e7b8c68bf89a18b3cc4316cc45c984aa6a5366d09bd5c6e3cd514344"} Jul 09 10:01:26.262820 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.262542 2571 scope.go:117] "RemoveContainer" containerID="e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0" Jul 09 10:01:26.272152 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.272118 2571 scope.go:117] "RemoveContainer" containerID="e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0" Jul 09 10:01:26.272418 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:26.272401 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0\": container with ID starting with e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0 not found: ID does not exist" containerID="e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0" Jul 09 10:01:26.272471 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.272427 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0"} err="failed to get container status \"e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0\": rpc error: code = NotFound desc = could not find container \"e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0\": container with ID starting with e928b1f1d0cb540898edee24dd1fa05c5df0e98bdf66cc6dacd46a33fe17fee0 not found: ID does not exist" Jul 09 10:01:26.293071 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.293041 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d85fbc597-gpxsd"] Jul 09 10:01:26.298360 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:26.298341 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d85fbc597-gpxsd"] Jul 09 10:01:27.160259 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:27.160228 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" path="/var/lib/kubelet/pods/7a0c84ed-49e0-470a-8ccc-a08f2cd5a499/volumes" Jul 09 10:01:36.367372 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.367331 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwldb"] Jul 09 10:01:36.367936 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.367829 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5d8503b-0b68-4f75-a9fe-117c651bdab9" containerName="manager" Jul 09 10:01:36.367936 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.367848 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5d8503b-0b68-4f75-a9fe-117c651bdab9" containerName="manager" Jul 09 10:01:36.367936 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.367879 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" containerName="console" Jul 09 10:01:36.367936 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.367887 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" containerName="console" Jul 09 10:01:36.368201 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.367977 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a0c84ed-49e0-470a-8ccc-a08f2cd5a499" containerName="console" Jul 09 10:01:36.368201 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.367989 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5d8503b-0b68-4f75-a9fe-117c651bdab9" containerName="manager" Jul 09 10:01:36.371099 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.371078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:36.373823 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.373800 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zvqhc\"" Jul 09 10:01:36.373942 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.373863 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 10:01:36.379415 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.379398 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwldb"] Jul 09 10:01:36.452518 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.452488 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nrj\" (UniqueName: \"kubernetes.io/projected/b8d68e59-95da-4362-837c-31bd086d23c0-kube-api-access-q2nrj\") pod \"limitador-limitador-64c94fd64f-lwldb\" (UID: \"b8d68e59-95da-4362-837c-31bd086d23c0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:36.452683 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.452533 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b8d68e59-95da-4362-837c-31bd086d23c0-config-file\") pod \"limitador-limitador-64c94fd64f-lwldb\" (UID: \"b8d68e59-95da-4362-837c-31bd086d23c0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:36.470017 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.469973 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwldb"] Jul 09 10:01:36.553855 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.553817 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nrj\" (UniqueName: \"kubernetes.io/projected/b8d68e59-95da-4362-837c-31bd086d23c0-kube-api-access-q2nrj\") pod \"limitador-limitador-64c94fd64f-lwldb\" (UID: \"b8d68e59-95da-4362-837c-31bd086d23c0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:36.554009 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.553862 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b8d68e59-95da-4362-837c-31bd086d23c0-config-file\") pod \"limitador-limitador-64c94fd64f-lwldb\" (UID: \"b8d68e59-95da-4362-837c-31bd086d23c0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:36.554469 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.554450 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b8d68e59-95da-4362-837c-31bd086d23c0-config-file\") pod \"limitador-limitador-64c94fd64f-lwldb\" (UID: \"b8d68e59-95da-4362-837c-31bd086d23c0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:36.564422 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.564398 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nrj\" (UniqueName: \"kubernetes.io/projected/b8d68e59-95da-4362-837c-31bd086d23c0-kube-api-access-q2nrj\") pod \"limitador-limitador-64c94fd64f-lwldb\" (UID: \"b8d68e59-95da-4362-837c-31bd086d23c0\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:36.682333 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.682247 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:36.822158 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:36.818710 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwldb"] Jul 09 10:01:36.824114 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:01:36.824084 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d68e59_95da_4362_837c_31bd086d23c0.slice/crio-8b54d588c861c6b6ad75d657cb66856215720a67095df048dafac22b6ac8a0e1 WatchSource:0}: Error finding container 8b54d588c861c6b6ad75d657cb66856215720a67095df048dafac22b6ac8a0e1: Status 404 returned error can't find the container with id 8b54d588c861c6b6ad75d657cb66856215720a67095df048dafac22b6ac8a0e1 Jul 09 10:01:37.219715 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.219683 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-dmjsx"] Jul 09 10:01:37.224627 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.224604 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dmjsx" Jul 09 10:01:37.227927 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.227906 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rvd7h\"" Jul 09 10:01:37.232353 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.232328 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dmjsx"] Jul 09 10:01:37.260047 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.260021 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwbc\" (UniqueName: \"kubernetes.io/projected/b85e3a9b-5e6a-4495-826d-2e003f3111eb-kube-api-access-lnwbc\") pod \"authorino-58cd955675-dmjsx\" (UID: \"b85e3a9b-5e6a-4495-826d-2e003f3111eb\") " pod="kuadrant-system/authorino-58cd955675-dmjsx" Jul 09 10:01:37.324000 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.323968 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" event={"ID":"b8d68e59-95da-4362-837c-31bd086d23c0","Type":"ContainerStarted","Data":"8b54d588c861c6b6ad75d657cb66856215720a67095df048dafac22b6ac8a0e1"} Jul 09 10:01:37.360893 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.360740 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwbc\" (UniqueName: \"kubernetes.io/projected/b85e3a9b-5e6a-4495-826d-2e003f3111eb-kube-api-access-lnwbc\") pod \"authorino-58cd955675-dmjsx\" (UID: \"b85e3a9b-5e6a-4495-826d-2e003f3111eb\") " pod="kuadrant-system/authorino-58cd955675-dmjsx" Jul 09 10:01:37.373375 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.373348 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwbc\" (UniqueName: \"kubernetes.io/projected/b85e3a9b-5e6a-4495-826d-2e003f3111eb-kube-api-access-lnwbc\") pod \"authorino-58cd955675-dmjsx\" (UID: \"b85e3a9b-5e6a-4495-826d-2e003f3111eb\") " pod="kuadrant-system/authorino-58cd955675-dmjsx" Jul 09 10:01:37.535577 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.535492 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dmjsx" Jul 09 10:01:37.726551 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:37.726518 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dmjsx"] Jul 09 10:01:37.728409 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:01:37.728311 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb85e3a9b_5e6a_4495_826d_2e003f3111eb.slice/crio-3ad10c7a45f11a88f1dce3f491fde89f9d09e9af2862b41bb3989b32aa4c2920 WatchSource:0}: Error finding container 3ad10c7a45f11a88f1dce3f491fde89f9d09e9af2862b41bb3989b32aa4c2920: Status 404 returned error can't find the container with id 3ad10c7a45f11a88f1dce3f491fde89f9d09e9af2862b41bb3989b32aa4c2920 Jul 09 10:01:38.330717 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:38.330685 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dmjsx" event={"ID":"b85e3a9b-5e6a-4495-826d-2e003f3111eb","Type":"ContainerStarted","Data":"3ad10c7a45f11a88f1dce3f491fde89f9d09e9af2862b41bb3989b32aa4c2920"} Jul 09 10:01:40.346751 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:40.346705 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" event={"ID":"b8d68e59-95da-4362-837c-31bd086d23c0","Type":"ContainerStarted","Data":"25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718"} Jul 09 10:01:40.347225 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:40.346830 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:40.370155 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:40.370080 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" podStartSLOduration=1.6598216049999999 podStartE2EDuration="4.370060699s" totalImagesPullingTime="2.710239094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:01:40.367666244 +0000 UTC m=+455.700150040" watchObservedRunningTime="2026-07-09 10:01:40.370060699 +0000 UTC m=+455.702544496" Jul 09 10:01:42.360301 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:42.360264 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dmjsx" event={"ID":"b85e3a9b-5e6a-4495-826d-2e003f3111eb","Type":"ContainerStarted","Data":"3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c"} Jul 09 10:01:42.379090 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:42.379041 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-dmjsx" podStartSLOduration=1.517408004 podStartE2EDuration="5.379026929s" totalImagesPullingTime="3.861618925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:01:42.377480569 +0000 UTC m=+457.709964367" watchObservedRunningTime="2026-07-09 10:01:42.379026929 +0000 UTC m=+457.711510726" Jul 09 10:01:42.896915 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:42.896876 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dmjsx"] Jul 09 10:01:44.372058 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:44.372021 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-dmjsx" podUID="b85e3a9b-5e6a-4495-826d-2e003f3111eb" containerName="authorino" containerID="cri-o://3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c" gracePeriod=30 Jul 09 10:01:44.619489 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:44.619463 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dmjsx" Jul 09 10:01:44.730770 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:44.730734 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnwbc\" (UniqueName: \"kubernetes.io/projected/b85e3a9b-5e6a-4495-826d-2e003f3111eb-kube-api-access-lnwbc\") pod \"b85e3a9b-5e6a-4495-826d-2e003f3111eb\" (UID: \"b85e3a9b-5e6a-4495-826d-2e003f3111eb\") " Jul 09 10:01:44.732785 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:44.732746 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85e3a9b-5e6a-4495-826d-2e003f3111eb-kube-api-access-lnwbc" (OuterVolumeSpecName: "kube-api-access-lnwbc") pod "b85e3a9b-5e6a-4495-826d-2e003f3111eb" (UID: "b85e3a9b-5e6a-4495-826d-2e003f3111eb"). InnerVolumeSpecName "kube-api-access-lnwbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:01:44.831890 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:44.831854 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnwbc\" (UniqueName: \"kubernetes.io/projected/b85e3a9b-5e6a-4495-826d-2e003f3111eb-kube-api-access-lnwbc\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:45.378713 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.378674 2571 generic.go:356] "Generic (PLEG): container finished" podID="b85e3a9b-5e6a-4495-826d-2e003f3111eb" containerID="3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c" exitCode=0 Jul 09 10:01:45.378713 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.378716 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dmjsx" Jul 09 10:01:45.379180 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.378756 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dmjsx" event={"ID":"b85e3a9b-5e6a-4495-826d-2e003f3111eb","Type":"ContainerDied","Data":"3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c"} Jul 09 10:01:45.379180 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.378801 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dmjsx" event={"ID":"b85e3a9b-5e6a-4495-826d-2e003f3111eb","Type":"ContainerDied","Data":"3ad10c7a45f11a88f1dce3f491fde89f9d09e9af2862b41bb3989b32aa4c2920"} Jul 09 10:01:45.379180 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.378821 2571 scope.go:117] "RemoveContainer" containerID="3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c" Jul 09 10:01:45.387627 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.387611 2571 scope.go:117] "RemoveContainer" containerID="3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c" Jul 09 10:01:45.387896 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:45.387874 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c\": container with ID starting with 3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c not found: ID does not exist" containerID="3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c" Jul 09 10:01:45.387950 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.387904 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c"} err="failed to get container status \"3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c\": rpc error: code = NotFound desc = could not find container \"3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c\": container with ID starting with 3f9a30acea8cf753aca315097c8308fa6ee6ccd3827d6a15021a885ddc01fd2c not found: ID does not exist" Jul 09 10:01:45.400771 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.400750 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dmjsx"] Jul 09 10:01:45.403345 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:45.403327 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-dmjsx"] Jul 09 10:01:47.160416 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:47.160381 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85e3a9b-5e6a-4495-826d-2e003f3111eb" path="/var/lib/kubelet/pods/b85e3a9b-5e6a-4495-826d-2e003f3111eb/volumes" Jul 09 10:01:51.353562 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:51.353533 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:51.411103 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:51.411031 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwldb"] Jul 09 10:01:51.411354 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:51.411317 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" podUID="b8d68e59-95da-4362-837c-31bd086d23c0" containerName="limitador" containerID="cri-o://25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718" gracePeriod=30 Jul 09 10:01:51.962804 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:51.962779 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:52.094741 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.094709 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2nrj\" (UniqueName: \"kubernetes.io/projected/b8d68e59-95da-4362-837c-31bd086d23c0-kube-api-access-q2nrj\") pod \"b8d68e59-95da-4362-837c-31bd086d23c0\" (UID: \"b8d68e59-95da-4362-837c-31bd086d23c0\") " Jul 09 10:01:52.094915 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.094768 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b8d68e59-95da-4362-837c-31bd086d23c0-config-file\") pod \"b8d68e59-95da-4362-837c-31bd086d23c0\" (UID: \"b8d68e59-95da-4362-837c-31bd086d23c0\") " Jul 09 10:01:52.095200 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.095174 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d68e59-95da-4362-837c-31bd086d23c0-config-file" (OuterVolumeSpecName: "config-file") pod "b8d68e59-95da-4362-837c-31bd086d23c0" (UID: "b8d68e59-95da-4362-837c-31bd086d23c0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:01:52.096771 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.096744 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d68e59-95da-4362-837c-31bd086d23c0-kube-api-access-q2nrj" (OuterVolumeSpecName: "kube-api-access-q2nrj") pod "b8d68e59-95da-4362-837c-31bd086d23c0" (UID: "b8d68e59-95da-4362-837c-31bd086d23c0"). InnerVolumeSpecName "kube-api-access-q2nrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:01:52.195737 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.195708 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nrj\" (UniqueName: \"kubernetes.io/projected/b8d68e59-95da-4362-837c-31bd086d23c0-kube-api-access-q2nrj\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:52.195737 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.195733 2571 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b8d68e59-95da-4362-837c-31bd086d23c0-config-file\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:01:52.417460 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.417375 2571 generic.go:356] "Generic (PLEG): container finished" podID="b8d68e59-95da-4362-837c-31bd086d23c0" containerID="25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718" exitCode=0 Jul 09 10:01:52.417460 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.417420 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" Jul 09 10:01:52.417931 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.417457 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" event={"ID":"b8d68e59-95da-4362-837c-31bd086d23c0","Type":"ContainerDied","Data":"25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718"} Jul 09 10:01:52.417931 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.417505 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-lwldb" event={"ID":"b8d68e59-95da-4362-837c-31bd086d23c0","Type":"ContainerDied","Data":"8b54d588c861c6b6ad75d657cb66856215720a67095df048dafac22b6ac8a0e1"} Jul 09 10:01:52.417931 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.417527 2571 scope.go:117] "RemoveContainer" containerID="25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718" Jul 09 10:01:52.427107 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.427092 2571 scope.go:117] "RemoveContainer" containerID="25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718" Jul 09 10:01:52.427369 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:01:52.427352 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718\": container with ID starting with 25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718 not found: ID does not exist" containerID="25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718" Jul 09 10:01:52.427418 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.427378 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718"} err="failed to get container status \"25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718\": rpc error: code = NotFound desc = could not find container \"25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718\": container with ID starting with 25fc5179b414f1fdc1dcc5ecedf580773c7b92078269dc111a5fc5e6f2618718 not found: ID does not exist" Jul 09 10:01:52.443059 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.443031 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwldb"] Jul 09 10:01:52.447577 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:52.447557 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-lwldb"] Jul 09 10:01:53.160721 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:53.160681 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d68e59-95da-4362-837c-31bd086d23c0" path="/var/lib/kubelet/pods/b8d68e59-95da-4362-837c-31bd086d23c0/volumes" Jul 09 10:01:58.025657 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.025600 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lrflc"] Jul 09 10:01:58.026202 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.026168 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d68e59-95da-4362-837c-31bd086d23c0" containerName="limitador" Jul 09 10:01:58.026202 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.026194 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d68e59-95da-4362-837c-31bd086d23c0" containerName="limitador" Jul 09 10:01:58.026362 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.026215 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85e3a9b-5e6a-4495-826d-2e003f3111eb" containerName="authorino" Jul 09 10:01:58.026362 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.026224 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85e3a9b-5e6a-4495-826d-2e003f3111eb" containerName="authorino" Jul 09 10:01:58.026362 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.026345 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85e3a9b-5e6a-4495-826d-2e003f3111eb" containerName="authorino" Jul 09 10:01:58.026362 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.026359 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d68e59-95da-4362-837c-31bd086d23c0" containerName="limitador" Jul 09 10:01:58.030905 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.030887 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:01:58.033803 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.033781 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 10:01:58.033803 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.033793 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 10:01:58.034090 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.034073 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 10:01:58.034904 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.034890 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kgktm\"" Jul 09 10:01:58.041677 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.041655 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lrflc"] Jul 09 10:01:58.147839 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.147803 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e70ed47-4021-4c63-b038-8c1400600b40-data\") pod \"postgres-69fb4594cf-lrflc\" (UID: \"5e70ed47-4021-4c63-b038-8c1400600b40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:01:58.148019 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.147855 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rndjs\" (UniqueName: \"kubernetes.io/projected/5e70ed47-4021-4c63-b038-8c1400600b40-kube-api-access-rndjs\") pod \"postgres-69fb4594cf-lrflc\" (UID: \"5e70ed47-4021-4c63-b038-8c1400600b40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:01:58.248813 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.248774 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e70ed47-4021-4c63-b038-8c1400600b40-data\") pod \"postgres-69fb4594cf-lrflc\" (UID: \"5e70ed47-4021-4c63-b038-8c1400600b40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:01:58.249000 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.248823 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndjs\" (UniqueName: \"kubernetes.io/projected/5e70ed47-4021-4c63-b038-8c1400600b40-kube-api-access-rndjs\") pod \"postgres-69fb4594cf-lrflc\" (UID: \"5e70ed47-4021-4c63-b038-8c1400600b40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:01:58.249203 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.249182 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5e70ed47-4021-4c63-b038-8c1400600b40-data\") pod \"postgres-69fb4594cf-lrflc\" (UID: \"5e70ed47-4021-4c63-b038-8c1400600b40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:01:58.258604 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.258571 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rndjs\" (UniqueName: \"kubernetes.io/projected/5e70ed47-4021-4c63-b038-8c1400600b40-kube-api-access-rndjs\") pod \"postgres-69fb4594cf-lrflc\" (UID: \"5e70ed47-4021-4c63-b038-8c1400600b40\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:01:58.342544 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.342513 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:01:58.481910 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:58.481884 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lrflc"] Jul 09 10:01:58.483397 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:01:58.483369 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e70ed47_4021_4c63_b038_8c1400600b40.slice/crio-da67203770daadf7b0c16ce64c6a9939835dc92a72c35bd53ea0cb00c1ed4cc3 WatchSource:0}: Error finding container da67203770daadf7b0c16ce64c6a9939835dc92a72c35bd53ea0cb00c1ed4cc3: Status 404 returned error can't find the container with id da67203770daadf7b0c16ce64c6a9939835dc92a72c35bd53ea0cb00c1ed4cc3 Jul 09 10:01:59.457849 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:01:59.457817 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" event={"ID":"5e70ed47-4021-4c63-b038-8c1400600b40","Type":"ContainerStarted","Data":"da67203770daadf7b0c16ce64c6a9939835dc92a72c35bd53ea0cb00c1ed4cc3"} Jul 09 10:02:04.488815 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:04.488785 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" event={"ID":"5e70ed47-4021-4c63-b038-8c1400600b40","Type":"ContainerStarted","Data":"f76eaa37e64e7b7fd7a0a65c58e96f2a29ccf342b8a550f6c01b025ec7bc868b"} Jul 09 10:02:04.489220 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:04.488911 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:02:04.509008 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:04.508956 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" podStartSLOduration=1.460585969 podStartE2EDuration="6.508939646s" totalImagesPullingTime="5.048353677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:04.506478209 +0000 UTC m=+479.838961998" watchObservedRunningTime="2026-07-09 10:02:04.508939646 +0000 UTC m=+479.841423434" Jul 09 10:02:10.523551 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:10.523518 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lrflc" Jul 09 10:02:11.088578 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.088550 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nbqg7"] Jul 09 10:02:11.093638 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.093617 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" Jul 09 10:02:11.096577 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.096550 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rvd7h\"" Jul 09 10:02:11.099304 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.099281 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nbqg7"] Jul 09 10:02:11.160545 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.160514 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw7kn\" (UniqueName: \"kubernetes.io/projected/3d1241df-760b-430b-bf6a-15afd6f30397-kube-api-access-sw7kn\") pod \"authorino-5d4b67c446-nbqg7\" (UID: \"3d1241df-760b-430b-bf6a-15afd6f30397\") " pod="kuadrant-system/authorino-5d4b67c446-nbqg7" Jul 09 10:02:11.261734 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.261699 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw7kn\" (UniqueName: \"kubernetes.io/projected/3d1241df-760b-430b-bf6a-15afd6f30397-kube-api-access-sw7kn\") pod \"authorino-5d4b67c446-nbqg7\" (UID: \"3d1241df-760b-430b-bf6a-15afd6f30397\") " pod="kuadrant-system/authorino-5d4b67c446-nbqg7" Jul 09 10:02:11.272414 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.272381 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw7kn\" (UniqueName: \"kubernetes.io/projected/3d1241df-760b-430b-bf6a-15afd6f30397-kube-api-access-sw7kn\") pod \"authorino-5d4b67c446-nbqg7\" (UID: \"3d1241df-760b-430b-bf6a-15afd6f30397\") " pod="kuadrant-system/authorino-5d4b67c446-nbqg7" Jul 09 10:02:11.337751 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.337713 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nbqg7"] Jul 09 10:02:11.337967 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.337955 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" Jul 09 10:02:11.372016 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.371986 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6778c86bdd-mt8xh"] Jul 09 10:02:11.376518 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.376483 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" Jul 09 10:02:11.392033 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.391467 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6778c86bdd-mt8xh"] Jul 09 10:02:11.463777 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.463746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfggk\" (UniqueName: \"kubernetes.io/projected/147b1e79-c11b-46bc-935e-85de6c8c3451-kube-api-access-zfggk\") pod \"authorino-6778c86bdd-mt8xh\" (UID: \"147b1e79-c11b-46bc-935e-85de6c8c3451\") " pod="kuadrant-system/authorino-6778c86bdd-mt8xh" Jul 09 10:02:11.478630 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.478608 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nbqg7"] Jul 09 10:02:11.480231 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:02:11.480198 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d1241df_760b_430b_bf6a_15afd6f30397.slice/crio-c6c0add950e4c74cadca564ad8854fc224c43659fc7d8773ffa046cbe19a2887 WatchSource:0}: Error finding container c6c0add950e4c74cadca564ad8854fc224c43659fc7d8773ffa046cbe19a2887: Status 404 returned error can't find the container with id c6c0add950e4c74cadca564ad8854fc224c43659fc7d8773ffa046cbe19a2887 Jul 09 10:02:11.526186 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.526156 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" event={"ID":"3d1241df-760b-430b-bf6a-15afd6f30397","Type":"ContainerStarted","Data":"c6c0add950e4c74cadca564ad8854fc224c43659fc7d8773ffa046cbe19a2887"} Jul 09 10:02:11.564295 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.564266 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfggk\" (UniqueName: \"kubernetes.io/projected/147b1e79-c11b-46bc-935e-85de6c8c3451-kube-api-access-zfggk\") pod \"authorino-6778c86bdd-mt8xh\" (UID: \"147b1e79-c11b-46bc-935e-85de6c8c3451\") " pod="kuadrant-system/authorino-6778c86bdd-mt8xh" Jul 09 10:02:11.577896 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.577869 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfggk\" (UniqueName: \"kubernetes.io/projected/147b1e79-c11b-46bc-935e-85de6c8c3451-kube-api-access-zfggk\") pod \"authorino-6778c86bdd-mt8xh\" (UID: \"147b1e79-c11b-46bc-935e-85de6c8c3451\") " pod="kuadrant-system/authorino-6778c86bdd-mt8xh" Jul 09 10:02:11.687398 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.687322 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" Jul 09 10:02:11.727339 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.727304 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6778c86bdd-mt8xh"] Jul 09 10:02:11.760499 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.760467 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7968f75844-9lhm7"] Jul 09 10:02:11.765535 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.765513 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:02:11.768109 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.768092 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 10:02:11.772076 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.772052 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7968f75844-9lhm7"] Jul 09 10:02:11.827645 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.827618 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6778c86bdd-mt8xh"] Jul 09 10:02:11.828642 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:02:11.828622 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod147b1e79_c11b_46bc_935e_85de6c8c3451.slice/crio-74ce417f29a86c89968da912ab4936b67f6a48f6b6e5cbb027a7011a4fed6b88 WatchSource:0}: Error finding container 74ce417f29a86c89968da912ab4936b67f6a48f6b6e5cbb027a7011a4fed6b88: Status 404 returned error can't find the container with id 74ce417f29a86c89968da912ab4936b67f6a48f6b6e5cbb027a7011a4fed6b88 Jul 09 10:02:11.866285 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.866257 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5zv\" (UniqueName: \"kubernetes.io/projected/7e508629-69b8-46bd-8095-d7d4650fc790-kube-api-access-tq5zv\") pod \"authorino-7968f75844-9lhm7\" (UID: \"7e508629-69b8-46bd-8095-d7d4650fc790\") " pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:02:11.866427 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.866306 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e508629-69b8-46bd-8095-d7d4650fc790-tls-cert\") pod \"authorino-7968f75844-9lhm7\" (UID: \"7e508629-69b8-46bd-8095-d7d4650fc790\") " pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:02:11.967523 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.967442 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e508629-69b8-46bd-8095-d7d4650fc790-tls-cert\") pod \"authorino-7968f75844-9lhm7\" (UID: \"7e508629-69b8-46bd-8095-d7d4650fc790\") " pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:02:11.967661 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.967530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5zv\" (UniqueName: \"kubernetes.io/projected/7e508629-69b8-46bd-8095-d7d4650fc790-kube-api-access-tq5zv\") pod \"authorino-7968f75844-9lhm7\" (UID: \"7e508629-69b8-46bd-8095-d7d4650fc790\") " pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:02:11.969758 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.969738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e508629-69b8-46bd-8095-d7d4650fc790-tls-cert\") pod \"authorino-7968f75844-9lhm7\" (UID: \"7e508629-69b8-46bd-8095-d7d4650fc790\") " pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:02:11.977395 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:11.977373 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5zv\" (UniqueName: \"kubernetes.io/projected/7e508629-69b8-46bd-8095-d7d4650fc790-kube-api-access-tq5zv\") pod \"authorino-7968f75844-9lhm7\" (UID: \"7e508629-69b8-46bd-8095-d7d4650fc790\") " pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:02:12.075543 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.075494 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:02:12.210513 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.210493 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7968f75844-9lhm7"] Jul 09 10:02:12.212056 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:02:12.212034 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e508629_69b8_46bd_8095_d7d4650fc790.slice/crio-998bbf5f82e02fe3a61e93f44bf2a6a40fc79bc6387508c914f3027921d1c8bb WatchSource:0}: Error finding container 998bbf5f82e02fe3a61e93f44bf2a6a40fc79bc6387508c914f3027921d1c8bb: Status 404 returned error can't find the container with id 998bbf5f82e02fe3a61e93f44bf2a6a40fc79bc6387508c914f3027921d1c8bb Jul 09 10:02:12.533420 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.533342 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" event={"ID":"3d1241df-760b-430b-bf6a-15afd6f30397","Type":"ContainerStarted","Data":"8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c"} Jul 09 10:02:12.533420 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.533378 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" podUID="3d1241df-760b-430b-bf6a-15afd6f30397" containerName="authorino" containerID="cri-o://8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c" gracePeriod=30 Jul 09 10:02:12.534709 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.534686 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" event={"ID":"147b1e79-c11b-46bc-935e-85de6c8c3451","Type":"ContainerStarted","Data":"74ce417f29a86c89968da912ab4936b67f6a48f6b6e5cbb027a7011a4fed6b88"} Jul 09 10:02:12.535861 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.535838 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7968f75844-9lhm7" event={"ID":"7e508629-69b8-46bd-8095-d7d4650fc790","Type":"ContainerStarted","Data":"998bbf5f82e02fe3a61e93f44bf2a6a40fc79bc6387508c914f3027921d1c8bb"} Jul 09 10:02:12.552302 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.552264 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" podStartSLOduration=0.838278618 podStartE2EDuration="1.552251862s" totalImagesPullingTime="713.973244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:12.550818381 +0000 UTC m=+487.883302178" watchObservedRunningTime="2026-07-09 10:02:12.552251862 +0000 UTC m=+487.884735659" Jul 09 10:02:12.811318 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.811289 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" Jul 09 10:02:12.876656 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.876629 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw7kn\" (UniqueName: \"kubernetes.io/projected/3d1241df-760b-430b-bf6a-15afd6f30397-kube-api-access-sw7kn\") pod \"3d1241df-760b-430b-bf6a-15afd6f30397\" (UID: \"3d1241df-760b-430b-bf6a-15afd6f30397\") " Jul 09 10:02:12.878630 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.878592 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1241df-760b-430b-bf6a-15afd6f30397-kube-api-access-sw7kn" (OuterVolumeSpecName: "kube-api-access-sw7kn") pod "3d1241df-760b-430b-bf6a-15afd6f30397" (UID: "3d1241df-760b-430b-bf6a-15afd6f30397"). InnerVolumeSpecName "kube-api-access-sw7kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:02:12.978256 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:12.978232 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw7kn\" (UniqueName: \"kubernetes.io/projected/3d1241df-760b-430b-bf6a-15afd6f30397-kube-api-access-sw7kn\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:02:13.542465 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.542384 2571 generic.go:356] "Generic (PLEG): container finished" podID="3d1241df-760b-430b-bf6a-15afd6f30397" containerID="8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c" exitCode=0 Jul 09 10:02:13.542465 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.542436 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" Jul 09 10:02:13.542465 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.542435 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" event={"ID":"3d1241df-760b-430b-bf6a-15afd6f30397","Type":"ContainerDied","Data":"8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c"} Jul 09 10:02:13.542975 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.542482 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-nbqg7" event={"ID":"3d1241df-760b-430b-bf6a-15afd6f30397","Type":"ContainerDied","Data":"c6c0add950e4c74cadca564ad8854fc224c43659fc7d8773ffa046cbe19a2887"} Jul 09 10:02:13.542975 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.542508 2571 scope.go:117] "RemoveContainer" containerID="8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c" Jul 09 10:02:13.544185 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.544160 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7968f75844-9lhm7" event={"ID":"7e508629-69b8-46bd-8095-d7d4650fc790","Type":"ContainerStarted","Data":"bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c"} Jul 09 10:02:13.551291 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.551273 2571 scope.go:117] "RemoveContainer" containerID="8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c" Jul 09 10:02:13.551547 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:02:13.551529 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c\": container with ID starting with 8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c not found: ID does not exist" containerID="8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c" Jul 09 10:02:13.551616 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.551554 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c"} err="failed to get container status \"8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c\": rpc error: code = NotFound desc = could not find container \"8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c\": container with ID starting with 8aeba847fdad2b49918afee8ed55b7a073c8894d1542ba7ebf4c140f1493822c not found: ID does not exist" Jul 09 10:02:13.563403 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.563375 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nbqg7"] Jul 09 10:02:13.568528 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.568507 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-nbqg7"] Jul 09 10:02:13.586470 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:13.586429 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7968f75844-9lhm7" podStartSLOduration=1.8754903139999999 podStartE2EDuration="2.586416012s" totalImagesPullingTime="710.925698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:13.585852675 +0000 UTC m=+488.918336472" watchObservedRunningTime="2026-07-09 10:02:13.586416012 +0000 UTC m=+488.918899816" Jul 09 10:02:15.160371 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:15.160339 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1241df-760b-430b-bf6a-15afd6f30397" path="/var/lib/kubelet/pods/3d1241df-760b-430b-bf6a-15afd6f30397/volumes" Jul 09 10:02:16.565628 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:16.565595 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" event={"ID":"147b1e79-c11b-46bc-935e-85de6c8c3451","Type":"ContainerStarted","Data":"1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747"} Jul 09 10:02:16.566020 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:16.565682 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" podUID="147b1e79-c11b-46bc-935e-85de6c8c3451" containerName="authorino" containerID="cri-o://1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747" gracePeriod=30 Jul 09 10:02:16.584034 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:16.583988 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" podStartSLOduration=1.035291339 podStartE2EDuration="5.583972244s" totalImagesPullingTime="4.548680905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:16.582225568 +0000 UTC m=+491.914709367" watchObservedRunningTime="2026-07-09 10:02:16.583972244 +0000 UTC m=+491.916456042" Jul 09 10:02:16.869209 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:16.869184 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" Jul 09 10:02:16.913338 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:16.913306 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfggk\" (UniqueName: \"kubernetes.io/projected/147b1e79-c11b-46bc-935e-85de6c8c3451-kube-api-access-zfggk\") pod \"147b1e79-c11b-46bc-935e-85de6c8c3451\" (UID: \"147b1e79-c11b-46bc-935e-85de6c8c3451\") " Jul 09 10:02:16.915317 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:16.915277 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/147b1e79-c11b-46bc-935e-85de6c8c3451-kube-api-access-zfggk" (OuterVolumeSpecName: "kube-api-access-zfggk") pod "147b1e79-c11b-46bc-935e-85de6c8c3451" (UID: "147b1e79-c11b-46bc-935e-85de6c8c3451"). InnerVolumeSpecName "kube-api-access-zfggk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:02:17.014863 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.014781 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfggk\" (UniqueName: \"kubernetes.io/projected/147b1e79-c11b-46bc-935e-85de6c8c3451-kube-api-access-zfggk\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:02:17.572432 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.572397 2571 generic.go:356] "Generic (PLEG): container finished" podID="147b1e79-c11b-46bc-935e-85de6c8c3451" containerID="1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747" exitCode=0 Jul 09 10:02:17.572839 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.572437 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" Jul 09 10:02:17.572839 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.572482 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" event={"ID":"147b1e79-c11b-46bc-935e-85de6c8c3451","Type":"ContainerDied","Data":"1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747"} Jul 09 10:02:17.572839 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.572514 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6778c86bdd-mt8xh" event={"ID":"147b1e79-c11b-46bc-935e-85de6c8c3451","Type":"ContainerDied","Data":"74ce417f29a86c89968da912ab4936b67f6a48f6b6e5cbb027a7011a4fed6b88"} Jul 09 10:02:17.572839 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.572532 2571 scope.go:117] "RemoveContainer" containerID="1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747" Jul 09 10:02:17.581370 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.581350 2571 scope.go:117] "RemoveContainer" containerID="1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747" Jul 09 10:02:17.581636 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:02:17.581615 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747\": container with ID starting with 1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747 not found: ID does not exist" containerID="1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747" Jul 09 10:02:17.581688 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.581644 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747"} err="failed to get container status \"1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747\": rpc error: code = NotFound desc = could not find container \"1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747\": container with ID starting with 1315040ccf09a6e33086313f2999262df79f37e9bc601f58eeddf7ef671a0747 not found: ID does not exist" Jul 09 10:02:17.591748 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.591722 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6778c86bdd-mt8xh"] Jul 09 10:02:17.596698 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:17.596677 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6778c86bdd-mt8xh"] Jul 09 10:02:18.074623 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.074592 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6897cb449c-hdmp9"] Jul 09 10:02:18.074991 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.074977 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="147b1e79-c11b-46bc-935e-85de6c8c3451" containerName="authorino" Jul 09 10:02:18.075038 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.074993 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="147b1e79-c11b-46bc-935e-85de6c8c3451" containerName="authorino" Jul 09 10:02:18.075038 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.075011 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d1241df-760b-430b-bf6a-15afd6f30397" containerName="authorino" Jul 09 10:02:18.075038 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.075017 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d1241df-760b-430b-bf6a-15afd6f30397" containerName="authorino" Jul 09 10:02:18.075178 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.075075 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d1241df-760b-430b-bf6a-15afd6f30397" containerName="authorino" Jul 09 10:02:18.075178 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.075083 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="147b1e79-c11b-46bc-935e-85de6c8c3451" containerName="authorino" Jul 09 10:02:18.077670 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.077650 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:18.080418 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.080397 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n9h7r\"" Jul 09 10:02:18.081059 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.081034 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 10:02:18.081177 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.081037 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 10:02:18.092518 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.092494 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6897cb449c-hdmp9"] Jul 09 10:02:18.131610 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.131578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxjxp\" (UniqueName: \"kubernetes.io/projected/0dcb9675-9337-4171-b079-6340b6a5d60b-kube-api-access-mxjxp\") pod \"maas-controller-6897cb449c-hdmp9\" (UID: \"0dcb9675-9337-4171-b079-6340b6a5d60b\") " pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:18.131775 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.131647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dcb9675-9337-4171-b079-6340b6a5d60b-webhook-cert\") pod \"maas-controller-6897cb449c-hdmp9\" (UID: \"0dcb9675-9337-4171-b079-6340b6a5d60b\") " pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:18.233037 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.233000 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dcb9675-9337-4171-b079-6340b6a5d60b-webhook-cert\") pod \"maas-controller-6897cb449c-hdmp9\" (UID: \"0dcb9675-9337-4171-b079-6340b6a5d60b\") " pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:18.233230 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.233071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxjxp\" (UniqueName: \"kubernetes.io/projected/0dcb9675-9337-4171-b079-6340b6a5d60b-kube-api-access-mxjxp\") pod \"maas-controller-6897cb449c-hdmp9\" (UID: \"0dcb9675-9337-4171-b079-6340b6a5d60b\") " pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:18.235458 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.235426 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dcb9675-9337-4171-b079-6340b6a5d60b-webhook-cert\") pod \"maas-controller-6897cb449c-hdmp9\" (UID: \"0dcb9675-9337-4171-b079-6340b6a5d60b\") " pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:18.245628 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.245599 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxjxp\" (UniqueName: \"kubernetes.io/projected/0dcb9675-9337-4171-b079-6340b6a5d60b-kube-api-access-mxjxp\") pod \"maas-controller-6897cb449c-hdmp9\" (UID: \"0dcb9675-9337-4171-b079-6340b6a5d60b\") " pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:18.271093 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.271062 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84494cff7b-fm9cw"] Jul 09 10:02:18.273494 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.273476 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:18.285138 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.285106 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84494cff7b-fm9cw"] Jul 09 10:02:18.334153 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.334034 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcd7c\" (UniqueName: \"kubernetes.io/projected/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-kube-api-access-vcd7c\") pod \"maas-controller-84494cff7b-fm9cw\" (UID: \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\") " pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:18.334306 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.334160 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-webhook-cert\") pod \"maas-controller-84494cff7b-fm9cw\" (UID: \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\") " pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:18.388066 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.388024 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:18.435270 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.435233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-webhook-cert\") pod \"maas-controller-84494cff7b-fm9cw\" (UID: \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\") " pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:18.435428 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.435362 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcd7c\" (UniqueName: \"kubernetes.io/projected/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-kube-api-access-vcd7c\") pod \"maas-controller-84494cff7b-fm9cw\" (UID: \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\") " pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:18.438428 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.438059 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-webhook-cert\") pod \"maas-controller-84494cff7b-fm9cw\" (UID: \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\") " pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:18.445588 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.445557 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcd7c\" (UniqueName: \"kubernetes.io/projected/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-kube-api-access-vcd7c\") pod \"maas-controller-84494cff7b-fm9cw\" (UID: \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\") " pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:18.522807 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.522780 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6897cb449c-hdmp9"] Jul 09 10:02:18.524691 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:02:18.524665 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcb9675_9337_4171_b079_6340b6a5d60b.slice/crio-27822cd908da5ac7804ea28159b7d0f27889bc5217edecb30568f0518f885f2b WatchSource:0}: Error finding container 27822cd908da5ac7804ea28159b7d0f27889bc5217edecb30568f0518f885f2b: Status 404 returned error can't find the container with id 27822cd908da5ac7804ea28159b7d0f27889bc5217edecb30568f0518f885f2b Jul 09 10:02:18.579186 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.579151 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6897cb449c-hdmp9" event={"ID":"0dcb9675-9337-4171-b079-6340b6a5d60b","Type":"ContainerStarted","Data":"27822cd908da5ac7804ea28159b7d0f27889bc5217edecb30568f0518f885f2b"} Jul 09 10:02:18.584413 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.584350 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:18.720688 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:18.720656 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84494cff7b-fm9cw"] Jul 09 10:02:18.726014 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:02:18.725988 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb45496a_61fb_4b26_aaeb_d6ecff1029a0.slice/crio-00deb53a99c9b429968eeb6f2768577b3fb6a7614880b34565ce3c8e9d8ca4ac WatchSource:0}: Error finding container 00deb53a99c9b429968eeb6f2768577b3fb6a7614880b34565ce3c8e9d8ca4ac: Status 404 returned error can't find the container with id 00deb53a99c9b429968eeb6f2768577b3fb6a7614880b34565ce3c8e9d8ca4ac Jul 09 10:02:19.159909 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:19.159875 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="147b1e79-c11b-46bc-935e-85de6c8c3451" path="/var/lib/kubelet/pods/147b1e79-c11b-46bc-935e-85de6c8c3451/volumes" Jul 09 10:02:19.590001 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:19.589960 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84494cff7b-fm9cw" event={"ID":"bb45496a-61fb-4b26-aaeb-d6ecff1029a0","Type":"ContainerStarted","Data":"00deb53a99c9b429968eeb6f2768577b3fb6a7614880b34565ce3c8e9d8ca4ac"} Jul 09 10:02:21.603192 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:21.603155 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84494cff7b-fm9cw" event={"ID":"bb45496a-61fb-4b26-aaeb-d6ecff1029a0","Type":"ContainerStarted","Data":"9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e"} Jul 09 10:02:21.603760 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:21.603246 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:21.604793 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:21.604769 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6897cb449c-hdmp9" event={"ID":"0dcb9675-9337-4171-b079-6340b6a5d60b","Type":"ContainerStarted","Data":"390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8"} Jul 09 10:02:21.604910 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:21.604896 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:21.626006 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:21.625961 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84494cff7b-fm9cw" podStartSLOduration=1.671050711 podStartE2EDuration="3.625947469s" totalImagesPullingTime="1.954896758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:21.624003916 +0000 UTC m=+496.956487733" watchObservedRunningTime="2026-07-09 10:02:21.625947469 +0000 UTC m=+496.958431285" Jul 09 10:02:21.644693 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:21.644637 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6897cb449c-hdmp9" podStartSLOduration=1.477445319 podStartE2EDuration="3.644619812s" totalImagesPullingTime="2.167174493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:21.642881061 +0000 UTC m=+496.975364859" watchObservedRunningTime="2026-07-09 10:02:21.644619812 +0000 UTC m=+496.977103611" Jul 09 10:02:32.617034 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:32.616998 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:02:32.617513 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:32.617326 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:32.685559 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:32.685522 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6897cb449c-hdmp9"] Jul 09 10:02:32.685787 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:32.685764 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6897cb449c-hdmp9" podUID="0dcb9675-9337-4171-b079-6340b6a5d60b" containerName="manager" containerID="cri-o://390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8" gracePeriod=30 Jul 09 10:02:32.934291 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:32.934267 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:33.063467 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.063436 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxjxp\" (UniqueName: \"kubernetes.io/projected/0dcb9675-9337-4171-b079-6340b6a5d60b-kube-api-access-mxjxp\") pod \"0dcb9675-9337-4171-b079-6340b6a5d60b\" (UID: \"0dcb9675-9337-4171-b079-6340b6a5d60b\") " Jul 09 10:02:33.063642 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.063486 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dcb9675-9337-4171-b079-6340b6a5d60b-webhook-cert\") pod \"0dcb9675-9337-4171-b079-6340b6a5d60b\" (UID: \"0dcb9675-9337-4171-b079-6340b6a5d60b\") " Jul 09 10:02:33.065600 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.065571 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcb9675-9337-4171-b079-6340b6a5d60b-kube-api-access-mxjxp" (OuterVolumeSpecName: "kube-api-access-mxjxp") pod "0dcb9675-9337-4171-b079-6340b6a5d60b" (UID: "0dcb9675-9337-4171-b079-6340b6a5d60b"). InnerVolumeSpecName "kube-api-access-mxjxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:02:33.065705 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.065613 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcb9675-9337-4171-b079-6340b6a5d60b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0dcb9675-9337-4171-b079-6340b6a5d60b" (UID: "0dcb9675-9337-4171-b079-6340b6a5d60b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:02:33.164318 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.164289 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxjxp\" (UniqueName: \"kubernetes.io/projected/0dcb9675-9337-4171-b079-6340b6a5d60b-kube-api-access-mxjxp\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:02:33.164318 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.164315 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0dcb9675-9337-4171-b079-6340b6a5d60b-webhook-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:02:33.672559 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.672529 2571 generic.go:356] "Generic (PLEG): container finished" podID="0dcb9675-9337-4171-b079-6340b6a5d60b" containerID="390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8" exitCode=0 Jul 09 10:02:33.672988 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.672566 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6897cb449c-hdmp9" Jul 09 10:02:33.672988 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.672571 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6897cb449c-hdmp9" event={"ID":"0dcb9675-9337-4171-b079-6340b6a5d60b","Type":"ContainerDied","Data":"390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8"} Jul 09 10:02:33.672988 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.672610 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6897cb449c-hdmp9" event={"ID":"0dcb9675-9337-4171-b079-6340b6a5d60b","Type":"ContainerDied","Data":"27822cd908da5ac7804ea28159b7d0f27889bc5217edecb30568f0518f885f2b"} Jul 09 10:02:33.672988 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.672632 2571 scope.go:117] "RemoveContainer" containerID="390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8" Jul 09 10:02:33.682239 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.682202 2571 scope.go:117] "RemoveContainer" containerID="390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8" Jul 09 10:02:33.682534 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:02:33.682504 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8\": container with ID starting with 390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8 not found: ID does not exist" containerID="390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8" Jul 09 10:02:33.682634 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.682540 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8"} err="failed to get container status \"390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8\": rpc error: code = NotFound desc = could not find container \"390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8\": container with ID starting with 390e88768abad877404de0471dc2072b558c1de040d5ae88beb6c5f9bd9b7ea8 not found: ID does not exist" Jul 09 10:02:33.707694 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.707666 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6897cb449c-hdmp9"] Jul 09 10:02:33.713362 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:33.712108 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6897cb449c-hdmp9"] Jul 09 10:02:35.160222 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:35.160190 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dcb9675-9337-4171-b079-6340b6a5d60b" path="/var/lib/kubelet/pods/0dcb9675-9337-4171-b079-6340b6a5d60b/volumes" Jul 09 10:02:51.863729 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.863695 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d47869698-f7mns"] Jul 09 10:02:51.864169 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.864069 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dcb9675-9337-4171-b079-6340b6a5d60b" containerName="manager" Jul 09 10:02:51.864169 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.864081 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dcb9675-9337-4171-b079-6340b6a5d60b" containerName="manager" Jul 09 10:02:51.864169 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.864164 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dcb9675-9337-4171-b079-6340b6a5d60b" containerName="manager" Jul 09 10:02:51.869424 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.869400 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:51.872138 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.872104 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 09 10:02:51.878579 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.878557 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d47869698-f7mns"] Jul 09 10:02:51.927421 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.927386 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c6dfcd-5fe7-4f59-ae44-e74080d74762-openshift-service-ca\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:51.927421 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.927425 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c6dfcd-5fe7-4f59-ae44-e74080d74762-maas-api-tls\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:51.927618 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:51.927467 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj6sc\" (UniqueName: \"kubernetes.io/projected/93c6dfcd-5fe7-4f59-ae44-e74080d74762-kube-api-access-jj6sc\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.028617 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.028577 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c6dfcd-5fe7-4f59-ae44-e74080d74762-openshift-service-ca\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.028617 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.028619 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c6dfcd-5fe7-4f59-ae44-e74080d74762-maas-api-tls\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.028826 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.028651 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj6sc\" (UniqueName: \"kubernetes.io/projected/93c6dfcd-5fe7-4f59-ae44-e74080d74762-kube-api-access-jj6sc\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.028826 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:02:52.028784 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 09 10:02:52.028899 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:02:52.028867 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93c6dfcd-5fe7-4f59-ae44-e74080d74762-maas-api-tls podName:93c6dfcd-5fe7-4f59-ae44-e74080d74762 nodeName:}" failed. No retries permitted until 2026-07-09 10:02:52.528845899 +0000 UTC m=+527.861329676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93c6dfcd-5fe7-4f59-ae44-e74080d74762-maas-api-tls") pod "maas-api-5d47869698-f7mns" (UID: "93c6dfcd-5fe7-4f59-ae44-e74080d74762") : secret "maas-api-serving-cert" not found Jul 09 10:02:52.029343 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.029327 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93c6dfcd-5fe7-4f59-ae44-e74080d74762-openshift-service-ca\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.038241 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.038218 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj6sc\" (UniqueName: \"kubernetes.io/projected/93c6dfcd-5fe7-4f59-ae44-e74080d74762-kube-api-access-jj6sc\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.412523 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.412484 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-q5nz8"] Jul 09 10:02:52.416087 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.416066 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:52.418855 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.418832 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 10:02:52.427437 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.427416 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-q5nz8"] Jul 09 10:02:52.431453 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.431432 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53671dad-c918-4117-8cee-a48b4a181ed8-plugins-config-volume\") pod \"payload-pre-processing-7964457696-q5nz8\" (UID: \"53671dad-c918-4117-8cee-a48b4a181ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:52.431562 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.431501 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thhkx\" (UniqueName: \"kubernetes.io/projected/53671dad-c918-4117-8cee-a48b4a181ed8-kube-api-access-thhkx\") pod \"payload-pre-processing-7964457696-q5nz8\" (UID: \"53671dad-c918-4117-8cee-a48b4a181ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:52.439442 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.439406 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xwmc9"] Jul 09 10:02:52.443180 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.443156 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:52.446215 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.446186 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vhktw\"" Jul 09 10:02:52.451707 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.451681 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xwmc9"] Jul 09 10:02:52.532079 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.531970 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c6dfcd-5fe7-4f59-ae44-e74080d74762-maas-api-tls\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.532297 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.532105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53671dad-c918-4117-8cee-a48b4a181ed8-plugins-config-volume\") pod \"payload-pre-processing-7964457696-q5nz8\" (UID: \"53671dad-c918-4117-8cee-a48b4a181ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:52.532297 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.532288 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thhkx\" (UniqueName: \"kubernetes.io/projected/53671dad-c918-4117-8cee-a48b4a181ed8-kube-api-access-thhkx\") pod \"payload-pre-processing-7964457696-q5nz8\" (UID: \"53671dad-c918-4117-8cee-a48b4a181ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:52.532427 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.532319 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d11b03-2e8c-4a47-b533-fbd3630ab03f-plugins-config-volume\") pod \"payload-processing-58f86c7798-xwmc9\" (UID: \"96d11b03-2e8c-4a47-b533-fbd3630ab03f\") " pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:52.532427 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.532346 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7prl\" (UniqueName: \"kubernetes.io/projected/96d11b03-2e8c-4a47-b533-fbd3630ab03f-kube-api-access-k7prl\") pod \"payload-processing-58f86c7798-xwmc9\" (UID: \"96d11b03-2e8c-4a47-b533-fbd3630ab03f\") " pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:52.533106 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.533080 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53671dad-c918-4117-8cee-a48b4a181ed8-plugins-config-volume\") pod \"payload-pre-processing-7964457696-q5nz8\" (UID: \"53671dad-c918-4117-8cee-a48b4a181ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:52.534936 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.534914 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93c6dfcd-5fe7-4f59-ae44-e74080d74762-maas-api-tls\") pod \"maas-api-5d47869698-f7mns\" (UID: \"93c6dfcd-5fe7-4f59-ae44-e74080d74762\") " pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.541348 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.541325 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thhkx\" (UniqueName: \"kubernetes.io/projected/53671dad-c918-4117-8cee-a48b4a181ed8-kube-api-access-thhkx\") pod \"payload-pre-processing-7964457696-q5nz8\" (UID: \"53671dad-c918-4117-8cee-a48b4a181ed8\") " pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:52.632797 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.632764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d11b03-2e8c-4a47-b533-fbd3630ab03f-plugins-config-volume\") pod \"payload-processing-58f86c7798-xwmc9\" (UID: \"96d11b03-2e8c-4a47-b533-fbd3630ab03f\") " pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:52.632797 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.632799 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7prl\" (UniqueName: \"kubernetes.io/projected/96d11b03-2e8c-4a47-b533-fbd3630ab03f-kube-api-access-k7prl\") pod \"payload-processing-58f86c7798-xwmc9\" (UID: \"96d11b03-2e8c-4a47-b533-fbd3630ab03f\") " pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:52.633569 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.633544 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96d11b03-2e8c-4a47-b533-fbd3630ab03f-plugins-config-volume\") pod \"payload-processing-58f86c7798-xwmc9\" (UID: \"96d11b03-2e8c-4a47-b533-fbd3630ab03f\") " pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:52.646736 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.646708 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7prl\" (UniqueName: \"kubernetes.io/projected/96d11b03-2e8c-4a47-b533-fbd3630ab03f-kube-api-access-k7prl\") pod \"payload-processing-58f86c7798-xwmc9\" (UID: \"96d11b03-2e8c-4a47-b533-fbd3630ab03f\") " pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:52.728971 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.728897 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:52.756026 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.755998 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:52.780994 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.780931 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:52.899108 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.898745 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-q5nz8"] Jul 09 10:02:52.901965 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:02:52.901932 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53671dad_c918_4117_8cee_a48b4a181ed8.slice/crio-8124cb7920c873838ffd90b4b9146a2c2058e1a12c26a049c3052abd12675025 WatchSource:0}: Error finding container 8124cb7920c873838ffd90b4b9146a2c2058e1a12c26a049c3052abd12675025: Status 404 returned error can't find the container with id 8124cb7920c873838ffd90b4b9146a2c2058e1a12c26a049c3052abd12675025 Jul 09 10:02:52.927925 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.927893 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xwmc9"] Jul 09 10:02:52.928203 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:02:52.928178 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d11b03_2e8c_4a47_b533_fbd3630ab03f.slice/crio-047fef98a1718409344fb51abbed2c1e53f6b7d44d7d4fcc4db8f5e6f7c4398c WatchSource:0}: Error finding container 047fef98a1718409344fb51abbed2c1e53f6b7d44d7d4fcc4db8f5e6f7c4398c: Status 404 returned error can't find the container with id 047fef98a1718409344fb51abbed2c1e53f6b7d44d7d4fcc4db8f5e6f7c4398c Jul 09 10:02:52.959999 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:52.959971 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d47869698-f7mns"] Jul 09 10:02:52.961470 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:02:52.961442 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c6dfcd_5fe7_4f59_ae44_e74080d74762.slice/crio-1cf85d16b5d74ca1326836308ade37eba7e24a952a65848563f38edafedcae13 WatchSource:0}: Error finding container 1cf85d16b5d74ca1326836308ade37eba7e24a952a65848563f38edafedcae13: Status 404 returned error can't find the container with id 1cf85d16b5d74ca1326836308ade37eba7e24a952a65848563f38edafedcae13 Jul 09 10:02:53.785423 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:53.785388 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" event={"ID":"96d11b03-2e8c-4a47-b533-fbd3630ab03f","Type":"ContainerStarted","Data":"047fef98a1718409344fb51abbed2c1e53f6b7d44d7d4fcc4db8f5e6f7c4398c"} Jul 09 10:02:53.786881 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:53.786851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" event={"ID":"93c6dfcd-5fe7-4f59-ae44-e74080d74762","Type":"ContainerStarted","Data":"1cf85d16b5d74ca1326836308ade37eba7e24a952a65848563f38edafedcae13"} Jul 09 10:02:53.788186 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:53.788157 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" event={"ID":"53671dad-c918-4117-8cee-a48b4a181ed8","Type":"ContainerStarted","Data":"8124cb7920c873838ffd90b4b9146a2c2058e1a12c26a049c3052abd12675025"} Jul 09 10:02:55.803149 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.803085 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" event={"ID":"96d11b03-2e8c-4a47-b533-fbd3630ab03f","Type":"ContainerStarted","Data":"f6dd6bcc6ad908d6541ebe0d14236331d67cbbba261db4b5981a8389cc4775f6"} Jul 09 10:02:55.803603 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.803217 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:02:55.804797 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.804771 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" event={"ID":"93c6dfcd-5fe7-4f59-ae44-e74080d74762","Type":"ContainerStarted","Data":"2c6b685d0f0ab7a57710a47eeb39adc7a37edbe46c01f324490adf724b626fef"} Jul 09 10:02:55.804925 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.804828 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:02:55.806375 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.806350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" event={"ID":"53671dad-c918-4117-8cee-a48b4a181ed8","Type":"ContainerStarted","Data":"a396d326f539f7ec8c9a00b2d1af2800164aa1f73607bc0d4343b614afe1ab0d"} Jul 09 10:02:55.806489 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.806446 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:02:55.823913 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.823870 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" podStartSLOduration=1.751022292 podStartE2EDuration="3.823856821s" totalImagesPullingTime="2.072834529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:55.820716315 +0000 UTC m=+531.153200126" watchObservedRunningTime="2026-07-09 10:02:55.823856821 +0000 UTC m=+531.156340653" Jul 09 10:02:55.839890 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.839839 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" podStartSLOduration=1.7431766770000001 podStartE2EDuration="3.83982252s" totalImagesPullingTime="2.096645843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:55.83952334 +0000 UTC m=+531.172007139" watchObservedRunningTime="2026-07-09 10:02:55.83982252 +0000 UTC m=+531.172306321" Jul 09 10:02:55.860233 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:02:55.860191 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" podStartSLOduration=2.822363429 podStartE2EDuration="4.860175157s" totalImagesPullingTime="2.037811728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:02:55.858199555 +0000 UTC m=+531.190683354" watchObservedRunningTime="2026-07-09 10:02:55.860175157 +0000 UTC m=+531.192658956" Jul 09 10:03:01.817238 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:01.817162 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d47869698-f7mns" Jul 09 10:03:06.814099 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:06.814064 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-xwmc9" Jul 09 10:03:06.814576 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:06.814466 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-q5nz8" Jul 09 10:03:16.079069 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.079035 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr"] Jul 09 10:03:16.084284 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.084263 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.088152 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.088109 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 10:03:16.088288 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.088153 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xzxkm\"" Jul 09 10:03:16.088288 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.088212 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 10:03:16.088288 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.088153 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 10:03:16.095930 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.095908 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr"] Jul 09 10:03:16.131769 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.131731 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.131903 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.131784 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8zz\" (UniqueName: \"kubernetes.io/projected/5b4cb26b-8238-4a95-9fa1-a343900369e1-kube-api-access-kd8zz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.131903 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.131843 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b4cb26b-8238-4a95-9fa1-a343900369e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.131984 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.131906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.131984 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.131931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.131984 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.131970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.232397 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.232366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8zz\" (UniqueName: \"kubernetes.io/projected/5b4cb26b-8238-4a95-9fa1-a343900369e1-kube-api-access-kd8zz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.232546 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.232407 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b4cb26b-8238-4a95-9fa1-a343900369e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.232590 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.232565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.232638 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.232622 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.232753 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.232735 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.232834 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.232797 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.233010 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.232984 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.233116 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.233092 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.233307 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.233286 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.234777 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.234759 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b4cb26b-8238-4a95-9fa1-a343900369e1-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.235014 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.234997 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b4cb26b-8238-4a95-9fa1-a343900369e1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.243888 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.243863 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8zz\" (UniqueName: \"kubernetes.io/projected/5b4cb26b-8238-4a95-9fa1-a343900369e1-kube-api-access-kd8zz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr\" (UID: \"5b4cb26b-8238-4a95-9fa1-a343900369e1\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.397103 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.397029 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:16.542719 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.542694 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr"] Jul 09 10:03:16.545570 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:03:16.545546 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4cb26b_8238_4a95_9fa1_a343900369e1.slice/crio-a54a9b66cc49ca2a272cc236c0c06900f4cda9304310212a4da93d653f5ca445 WatchSource:0}: Error finding container a54a9b66cc49ca2a272cc236c0c06900f4cda9304310212a4da93d653f5ca445: Status 404 returned error can't find the container with id a54a9b66cc49ca2a272cc236c0c06900f4cda9304310212a4da93d653f5ca445 Jul 09 10:03:16.928083 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:16.928040 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" event={"ID":"5b4cb26b-8238-4a95-9fa1-a343900369e1","Type":"ContainerStarted","Data":"a54a9b66cc49ca2a272cc236c0c06900f4cda9304310212a4da93d653f5ca445"} Jul 09 10:03:17.936152 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:17.936098 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" event={"ID":"5b4cb26b-8238-4a95-9fa1-a343900369e1","Type":"ContainerStarted","Data":"2d853f068de2b3fdb1a0cbe097aaec5f8843edcb61b79d1ada69a4142f09aae4"} Jul 09 10:03:17.956621 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:17.956562 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" podStartSLOduration=0.646981006 podStartE2EDuration="1.956544302s" totalImagesPullingTime="1.309563296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:03:17.955471069 +0000 UTC m=+553.287954868" watchObservedRunningTime="2026-07-09 10:03:17.956544302 +0000 UTC m=+553.289028100" Jul 09 10:03:26.397787 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:26.397753 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:26.398278 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:26.397985 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:26.410505 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:26.410484 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:27.005084 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:27.005051 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr" Jul 09 10:03:32.377050 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.377019 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4"] Jul 09 10:03:32.380767 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.380746 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.383914 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.383886 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 09 10:03:32.390735 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.390711 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4"] Jul 09 10:03:32.392951 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.392929 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.393189 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.393166 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.393329 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.393312 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836f8679-f5bd-468c-a6e9-f20abf391d9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.393456 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.393440 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.393587 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.393570 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.393713 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.393690 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8266\" (UniqueName: \"kubernetes.io/projected/836f8679-f5bd-468c-a6e9-f20abf391d9b-kube-api-access-c8266\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.494305 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8266\" (UniqueName: \"kubernetes.io/projected/836f8679-f5bd-468c-a6e9-f20abf391d9b-kube-api-access-c8266\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.494471 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494334 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.494471 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494381 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.494471 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836f8679-f5bd-468c-a6e9-f20abf391d9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.494471 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494412 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.494670 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494647 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.494855 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494834 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.494939 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494915 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.495012 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.494993 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.496817 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.496793 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/836f8679-f5bd-468c-a6e9-f20abf391d9b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.497104 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.497087 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/836f8679-f5bd-468c-a6e9-f20abf391d9b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.503390 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.503371 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8266\" (UniqueName: \"kubernetes.io/projected/836f8679-f5bd-468c-a6e9-f20abf391d9b-kube-api-access-c8266\") pod \"e2e-trlp-test-simulated-kserve-598466548d-lnqx4\" (UID: \"836f8679-f5bd-468c-a6e9-f20abf391d9b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.694425 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.694350 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:32.833623 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:32.833597 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4"] Jul 09 10:03:32.835553 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:03:32.835522 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836f8679_f5bd_468c_a6e9_f20abf391d9b.slice/crio-5438380aff4ecc2f3daae9020a67283d59ec8b8ee4d30a541c5238e671d4f58a WatchSource:0}: Error finding container 5438380aff4ecc2f3daae9020a67283d59ec8b8ee4d30a541c5238e671d4f58a: Status 404 returned error can't find the container with id 5438380aff4ecc2f3daae9020a67283d59ec8b8ee4d30a541c5238e671d4f58a Jul 09 10:03:33.027832 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:33.027746 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" event={"ID":"836f8679-f5bd-468c-a6e9-f20abf391d9b","Type":"ContainerStarted","Data":"5438380aff4ecc2f3daae9020a67283d59ec8b8ee4d30a541c5238e671d4f58a"} Jul 09 10:03:38.059592 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:38.059551 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" event={"ID":"836f8679-f5bd-468c-a6e9-f20abf391d9b","Type":"ContainerStarted","Data":"21ffe42921338a7fc2604f628c17516420b1a83537da94d9fb895f2b392917db"} Jul 09 10:03:38.089615 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:38.089568 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" podStartSLOduration=1.7588380670000001 podStartE2EDuration="6.089552845s" totalImagesPullingTime="4.330714778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:03:38.086963931 +0000 UTC m=+573.419447730" watchObservedRunningTime="2026-07-09 10:03:38.089552845 +0000 UTC m=+573.422036643" Jul 09 10:03:42.695375 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:42.695341 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:42.695739 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:42.695422 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:42.707827 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:42.707805 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:03:43.097882 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:03:43.097857 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-lnqx4" Jul 09 10:04:05.079894 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:05.079867 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:04:05.080589 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:05.080547 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:04:39.598567 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:39.598535 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7968f75844-9lhm7"] Jul 09 10:04:39.601401 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:39.598719 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7968f75844-9lhm7" podUID="7e508629-69b8-46bd-8095-d7d4650fc790" containerName="authorino" containerID="cri-o://bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c" gracePeriod=30 Jul 09 10:04:39.875893 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:39.875866 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:04:40.011308 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.011271 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq5zv\" (UniqueName: \"kubernetes.io/projected/7e508629-69b8-46bd-8095-d7d4650fc790-kube-api-access-tq5zv\") pod \"7e508629-69b8-46bd-8095-d7d4650fc790\" (UID: \"7e508629-69b8-46bd-8095-d7d4650fc790\") " Jul 09 10:04:40.011486 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.011356 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e508629-69b8-46bd-8095-d7d4650fc790-tls-cert\") pod \"7e508629-69b8-46bd-8095-d7d4650fc790\" (UID: \"7e508629-69b8-46bd-8095-d7d4650fc790\") " Jul 09 10:04:40.013467 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.013433 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e508629-69b8-46bd-8095-d7d4650fc790-kube-api-access-tq5zv" (OuterVolumeSpecName: "kube-api-access-tq5zv") pod "7e508629-69b8-46bd-8095-d7d4650fc790" (UID: "7e508629-69b8-46bd-8095-d7d4650fc790"). InnerVolumeSpecName "kube-api-access-tq5zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:04:40.022099 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.022074 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e508629-69b8-46bd-8095-d7d4650fc790-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7e508629-69b8-46bd-8095-d7d4650fc790" (UID: "7e508629-69b8-46bd-8095-d7d4650fc790"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:04:40.112457 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.112380 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq5zv\" (UniqueName: \"kubernetes.io/projected/7e508629-69b8-46bd-8095-d7d4650fc790-kube-api-access-tq5zv\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:04:40.112457 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.112408 2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e508629-69b8-46bd-8095-d7d4650fc790-tls-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:04:40.399243 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.399151 2571 generic.go:356] "Generic (PLEG): container finished" podID="7e508629-69b8-46bd-8095-d7d4650fc790" containerID="bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c" exitCode=0 Jul 09 10:04:40.399243 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.399193 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7968f75844-9lhm7" Jul 09 10:04:40.399457 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.399235 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7968f75844-9lhm7" event={"ID":"7e508629-69b8-46bd-8095-d7d4650fc790","Type":"ContainerDied","Data":"bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c"} Jul 09 10:04:40.399457 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.399282 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7968f75844-9lhm7" event={"ID":"7e508629-69b8-46bd-8095-d7d4650fc790","Type":"ContainerDied","Data":"998bbf5f82e02fe3a61e93f44bf2a6a40fc79bc6387508c914f3027921d1c8bb"} Jul 09 10:04:40.399457 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.399303 2571 scope.go:117] "RemoveContainer" containerID="bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c" Jul 09 10:04:40.408543 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.408524 2571 scope.go:117] "RemoveContainer" containerID="bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c" Jul 09 10:04:40.408770 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:04:40.408752 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c\": container with ID starting with bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c not found: ID does not exist" containerID="bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c" Jul 09 10:04:40.408814 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.408781 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c"} err="failed to get container status \"bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c\": rpc error: code = NotFound desc = could not find container \"bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c\": container with ID starting with bdc62e414b826304b9ba8d7295e96c0896554084d6e489cce4e2abff48e9342c not found: ID does not exist" Jul 09 10:04:40.423532 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.423501 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7968f75844-9lhm7"] Jul 09 10:04:40.428439 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:40.428417 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7968f75844-9lhm7"] Jul 09 10:04:41.161136 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:04:41.161086 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e508629-69b8-46bd-8095-d7d4650fc790" path="/var/lib/kubelet/pods/7e508629-69b8-46bd-8095-d7d4650fc790/volumes" Jul 09 10:06:10.107159 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.107048 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84494cff7b-fm9cw"] Jul 09 10:06:10.107740 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.107364 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-84494cff7b-fm9cw" podUID="bb45496a-61fb-4b26-aaeb-d6ecff1029a0" containerName="manager" containerID="cri-o://9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e" gracePeriod=30 Jul 09 10:06:10.370024 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.369958 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:06:10.410738 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.410704 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcd7c\" (UniqueName: \"kubernetes.io/projected/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-kube-api-access-vcd7c\") pod \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\" (UID: \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\") " Jul 09 10:06:10.410916 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.410821 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-webhook-cert\") pod \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\" (UID: \"bb45496a-61fb-4b26-aaeb-d6ecff1029a0\") " Jul 09 10:06:10.412841 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.412810 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bb45496a-61fb-4b26-aaeb-d6ecff1029a0" (UID: "bb45496a-61fb-4b26-aaeb-d6ecff1029a0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:06:10.412992 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.412974 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-kube-api-access-vcd7c" (OuterVolumeSpecName: "kube-api-access-vcd7c") pod "bb45496a-61fb-4b26-aaeb-d6ecff1029a0" (UID: "bb45496a-61fb-4b26-aaeb-d6ecff1029a0"). InnerVolumeSpecName "kube-api-access-vcd7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:06:10.511834 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.511807 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-webhook-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:06:10.511834 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.511831 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcd7c\" (UniqueName: \"kubernetes.io/projected/bb45496a-61fb-4b26-aaeb-d6ecff1029a0-kube-api-access-vcd7c\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:06:10.896568 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.896534 2571 generic.go:356] "Generic (PLEG): container finished" podID="bb45496a-61fb-4b26-aaeb-d6ecff1029a0" containerID="9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e" exitCode=0 Jul 09 10:06:10.896735 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.896579 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84494cff7b-fm9cw" Jul 09 10:06:10.896735 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.896607 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84494cff7b-fm9cw" event={"ID":"bb45496a-61fb-4b26-aaeb-d6ecff1029a0","Type":"ContainerDied","Data":"9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e"} Jul 09 10:06:10.896735 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.896655 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84494cff7b-fm9cw" event={"ID":"bb45496a-61fb-4b26-aaeb-d6ecff1029a0","Type":"ContainerDied","Data":"00deb53a99c9b429968eeb6f2768577b3fb6a7614880b34565ce3c8e9d8ca4ac"} Jul 09 10:06:10.896735 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.896684 2571 scope.go:117] "RemoveContainer" containerID="9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e" Jul 09 10:06:10.906688 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.906668 2571 scope.go:117] "RemoveContainer" containerID="9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e" Jul 09 10:06:10.906994 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:06:10.906962 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e\": container with ID starting with 9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e not found: ID does not exist" containerID="9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e" Jul 09 10:06:10.907056 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.907009 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e"} err="failed to get container status \"9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e\": rpc error: code = NotFound desc = could not find container \"9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e\": container with ID starting with 9e2a3c26f423e00dcb8113928fe46984ba5fdf20a14e928266748b4c50a9034e not found: ID does not exist" Jul 09 10:06:10.921856 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.921832 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84494cff7b-fm9cw"] Jul 09 10:06:10.928447 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:10.928426 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84494cff7b-fm9cw"] Jul 09 10:06:11.160933 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.160855 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb45496a-61fb-4b26-aaeb-d6ecff1029a0" path="/var/lib/kubelet/pods/bb45496a-61fb-4b26-aaeb-d6ecff1029a0/volumes" Jul 09 10:06:11.718054 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.718018 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84494cff7b-c997q"] Jul 09 10:06:11.718440 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.718427 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e508629-69b8-46bd-8095-d7d4650fc790" containerName="authorino" Jul 09 10:06:11.718486 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.718442 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e508629-69b8-46bd-8095-d7d4650fc790" containerName="authorino" Jul 09 10:06:11.718486 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.718461 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb45496a-61fb-4b26-aaeb-d6ecff1029a0" containerName="manager" Jul 09 10:06:11.718486 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.718466 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb45496a-61fb-4b26-aaeb-d6ecff1029a0" containerName="manager" Jul 09 10:06:11.718584 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.718526 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb45496a-61fb-4b26-aaeb-d6ecff1029a0" containerName="manager" Jul 09 10:06:11.718584 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.718538 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e508629-69b8-46bd-8095-d7d4650fc790" containerName="authorino" Jul 09 10:06:11.722835 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.722812 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:11.726648 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.726622 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 09 10:06:11.726648 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.726637 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 09 10:06:11.726822 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.726636 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n9h7r\"" Jul 09 10:06:11.732636 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.732615 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84494cff7b-c997q"] Jul 09 10:06:11.822438 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.822398 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49dda3df-d2e4-40e4-9fe2-e96094e6d743-webhook-cert\") pod \"maas-controller-84494cff7b-c997q\" (UID: \"49dda3df-d2e4-40e4-9fe2-e96094e6d743\") " pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:11.822596 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.822504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq2ml\" (UniqueName: \"kubernetes.io/projected/49dda3df-d2e4-40e4-9fe2-e96094e6d743-kube-api-access-wq2ml\") pod \"maas-controller-84494cff7b-c997q\" (UID: \"49dda3df-d2e4-40e4-9fe2-e96094e6d743\") " pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:11.923508 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.923476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49dda3df-d2e4-40e4-9fe2-e96094e6d743-webhook-cert\") pod \"maas-controller-84494cff7b-c997q\" (UID: \"49dda3df-d2e4-40e4-9fe2-e96094e6d743\") " pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:11.923667 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.923535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq2ml\" (UniqueName: \"kubernetes.io/projected/49dda3df-d2e4-40e4-9fe2-e96094e6d743-kube-api-access-wq2ml\") pod \"maas-controller-84494cff7b-c997q\" (UID: \"49dda3df-d2e4-40e4-9fe2-e96094e6d743\") " pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:11.925975 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.925948 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49dda3df-d2e4-40e4-9fe2-e96094e6d743-webhook-cert\") pod \"maas-controller-84494cff7b-c997q\" (UID: \"49dda3df-d2e4-40e4-9fe2-e96094e6d743\") " pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:11.933872 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:11.933845 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq2ml\" (UniqueName: \"kubernetes.io/projected/49dda3df-d2e4-40e4-9fe2-e96094e6d743-kube-api-access-wq2ml\") pod \"maas-controller-84494cff7b-c997q\" (UID: \"49dda3df-d2e4-40e4-9fe2-e96094e6d743\") " pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:12.034044 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:12.033958 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:12.179614 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:12.179592 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 10:06:12.180014 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:12.179874 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84494cff7b-c997q"] Jul 09 10:06:12.909634 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:12.909544 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84494cff7b-c997q" event={"ID":"49dda3df-d2e4-40e4-9fe2-e96094e6d743","Type":"ContainerStarted","Data":"adfe3947ed491c213b7d075eccdd5fcd18c0a03bd4234caf6a2c4574573569d7"} Jul 09 10:06:12.909634 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:12.909605 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84494cff7b-c997q" event={"ID":"49dda3df-d2e4-40e4-9fe2-e96094e6d743","Type":"ContainerStarted","Data":"93ca5d4748593e3a0d08caeb464a4ac6c2af823f2874ed3cef854cfe093560c6"} Jul 09 10:06:12.909634 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:12.909635 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:06:12.932202 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:12.932156 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84494cff7b-c997q" podStartSLOduration=1.466452241 podStartE2EDuration="1.932143162s" totalImagesPullingTime="465.690921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:06:12.930002571 +0000 UTC m=+728.262486368" watchObservedRunningTime="2026-07-09 10:06:12.932143162 +0000 UTC m=+728.264626956" Jul 09 10:06:23.923272 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:06:23.923238 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84494cff7b-c997q" Jul 09 10:09:05.115709 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:09:05.115632 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:09:05.119111 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:09:05.119088 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:10:56.233383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:56.233349 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt"] Jul 09 10:10:56.235914 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:56.233535 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" podUID="c33dcc2b-9100-48cd-a018-f1aff926cce6" containerName="manager" containerID="cri-o://a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231" gracePeriod=10 Jul 09 10:10:59.691963 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:59.691940 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:10:59.796101 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:59.796023 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33dcc2b-9100-48cd-a018-f1aff926cce6-extensions-socket-volume\") pod \"c33dcc2b-9100-48cd-a018-f1aff926cce6\" (UID: \"c33dcc2b-9100-48cd-a018-f1aff926cce6\") " Jul 09 10:10:59.796101 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:59.796068 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkzms\" (UniqueName: \"kubernetes.io/projected/c33dcc2b-9100-48cd-a018-f1aff926cce6-kube-api-access-jkzms\") pod \"c33dcc2b-9100-48cd-a018-f1aff926cce6\" (UID: \"c33dcc2b-9100-48cd-a018-f1aff926cce6\") " Jul 09 10:10:59.796526 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:59.796496 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33dcc2b-9100-48cd-a018-f1aff926cce6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c33dcc2b-9100-48cd-a018-f1aff926cce6" (UID: "c33dcc2b-9100-48cd-a018-f1aff926cce6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:10:59.798159 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:59.798136 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33dcc2b-9100-48cd-a018-f1aff926cce6-kube-api-access-jkzms" (OuterVolumeSpecName: "kube-api-access-jkzms") pod "c33dcc2b-9100-48cd-a018-f1aff926cce6" (UID: "c33dcc2b-9100-48cd-a018-f1aff926cce6"). InnerVolumeSpecName "kube-api-access-jkzms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:10:59.897041 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:59.897006 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c33dcc2b-9100-48cd-a018-f1aff926cce6-extensions-socket-volume\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:10:59.897041 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:10:59.897036 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkzms\" (UniqueName: \"kubernetes.io/projected/c33dcc2b-9100-48cd-a018-f1aff926cce6-kube-api-access-jkzms\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:11:00.500078 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.500046 2571 generic.go:356] "Generic (PLEG): container finished" podID="c33dcc2b-9100-48cd-a018-f1aff926cce6" containerID="a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231" exitCode=0 Jul 09 10:11:00.500304 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.500085 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" event={"ID":"c33dcc2b-9100-48cd-a018-f1aff926cce6","Type":"ContainerDied","Data":"a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231"} Jul 09 10:11:00.500304 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.500091 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" Jul 09 10:11:00.500304 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.500139 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt" event={"ID":"c33dcc2b-9100-48cd-a018-f1aff926cce6","Type":"ContainerDied","Data":"05a8fb6f060ad2ce48c48136544c4ee7208c63ed657dff394be1b7ee77e97fac"} Jul 09 10:11:00.500304 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.500157 2571 scope.go:117] "RemoveContainer" containerID="a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231" Jul 09 10:11:00.512427 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.512394 2571 scope.go:117] "RemoveContainer" containerID="a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231" Jul 09 10:11:00.513041 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:11:00.513003 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231\": container with ID starting with a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231 not found: ID does not exist" containerID="a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231" Jul 09 10:11:00.513184 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.513048 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231"} err="failed to get container status \"a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231\": rpc error: code = NotFound desc = could not find container \"a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231\": container with ID starting with a0b3dfe6bfae5e42eb870c681bc217fc7540e1c9d808731a095ef485e45e7231 not found: ID does not exist" Jul 09 10:11:00.525810 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.525786 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt"] Jul 09 10:11:00.532951 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:00.532918 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b8pxt"] Jul 09 10:11:01.160252 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:01.160220 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c33dcc2b-9100-48cd-a018-f1aff926cce6" path="/var/lib/kubelet/pods/c33dcc2b-9100-48cd-a018-f1aff926cce6/volumes" Jul 09 10:11:57.512707 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.512624 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b"] Jul 09 10:11:57.513218 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.513008 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c33dcc2b-9100-48cd-a018-f1aff926cce6" containerName="manager" Jul 09 10:11:57.513218 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.513019 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c33dcc2b-9100-48cd-a018-f1aff926cce6" containerName="manager" Jul 09 10:11:57.513218 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.513087 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c33dcc2b-9100-48cd-a018-f1aff926cce6" containerName="manager" Jul 09 10:11:57.516167 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.516150 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:57.518850 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.518832 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zc2xd\"" Jul 09 10:11:57.529310 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.529289 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b"] Jul 09 10:11:57.694156 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.694091 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db57175f-54e5-49ee-b451-130dbab8b490-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8z96b\" (UID: \"db57175f-54e5-49ee-b451-130dbab8b490\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:57.694342 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.694234 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gwc\" (UniqueName: \"kubernetes.io/projected/db57175f-54e5-49ee-b451-130dbab8b490-kube-api-access-h6gwc\") pod \"kuadrant-operator-controller-manager-86878c769f-8z96b\" (UID: \"db57175f-54e5-49ee-b451-130dbab8b490\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:57.794940 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.794857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gwc\" (UniqueName: \"kubernetes.io/projected/db57175f-54e5-49ee-b451-130dbab8b490-kube-api-access-h6gwc\") pod \"kuadrant-operator-controller-manager-86878c769f-8z96b\" (UID: \"db57175f-54e5-49ee-b451-130dbab8b490\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:57.794940 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.794924 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db57175f-54e5-49ee-b451-130dbab8b490-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8z96b\" (UID: \"db57175f-54e5-49ee-b451-130dbab8b490\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:57.795275 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.795254 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db57175f-54e5-49ee-b451-130dbab8b490-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8z96b\" (UID: \"db57175f-54e5-49ee-b451-130dbab8b490\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:57.805325 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.805300 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gwc\" (UniqueName: \"kubernetes.io/projected/db57175f-54e5-49ee-b451-130dbab8b490-kube-api-access-h6gwc\") pod \"kuadrant-operator-controller-manager-86878c769f-8z96b\" (UID: \"db57175f-54e5-49ee-b451-130dbab8b490\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:57.826276 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.826243 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:57.967196 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:57.967155 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b"] Jul 09 10:11:57.970702 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:11:57.970656 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb57175f_54e5_49ee_b451_130dbab8b490.slice/crio-d8745aa605ca58ac22fb7c73cd5fe90755503e303db552c49d42a8cd4f6fd6b0 WatchSource:0}: Error finding container d8745aa605ca58ac22fb7c73cd5fe90755503e303db552c49d42a8cd4f6fd6b0: Status 404 returned error can't find the container with id d8745aa605ca58ac22fb7c73cd5fe90755503e303db552c49d42a8cd4f6fd6b0 Jul 09 10:11:58.811082 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:58.811041 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" event={"ID":"db57175f-54e5-49ee-b451-130dbab8b490","Type":"ContainerStarted","Data":"a21c578ae498c890033c5f2e1e719580ed4cd2a33057e904b7836dbc8afa59ae"} Jul 09 10:11:58.811477 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:58.811106 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" event={"ID":"db57175f-54e5-49ee-b451-130dbab8b490","Type":"ContainerStarted","Data":"d8745aa605ca58ac22fb7c73cd5fe90755503e303db552c49d42a8cd4f6fd6b0"} Jul 09 10:11:58.811477 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:58.811247 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:11:58.834380 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:11:58.834329 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" podStartSLOduration=1.8343124880000001 podStartE2EDuration="1.834312488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:11:58.831206699 +0000 UTC m=+1074.163690497" watchObservedRunningTime="2026-07-09 10:11:58.834312488 +0000 UTC m=+1074.166796351" Jul 09 10:12:09.819304 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:12:09.819269 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8z96b" Jul 09 10:14:05.155096 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:14:05.155062 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:14:05.156886 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:14:05.156866 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:15:00.175631 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.175545 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8"] Jul 09 10:15:00.179114 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.179097 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" Jul 09 10:15:00.182183 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.182153 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npsbz\"" Jul 09 10:15:00.185642 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.185617 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8"] Jul 09 10:15:00.199303 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.199279 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w96lw\" (UniqueName: \"kubernetes.io/projected/c96338a8-3df3-451a-9371-124e3f270f5f-kube-api-access-w96lw\") pod \"maas-api-key-cleanup-29726535-67rs8\" (UID: \"c96338a8-3df3-451a-9371-124e3f270f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" Jul 09 10:15:00.300195 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.300153 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w96lw\" (UniqueName: \"kubernetes.io/projected/c96338a8-3df3-451a-9371-124e3f270f5f-kube-api-access-w96lw\") pod \"maas-api-key-cleanup-29726535-67rs8\" (UID: \"c96338a8-3df3-451a-9371-124e3f270f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" Jul 09 10:15:00.310278 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.310248 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w96lw\" (UniqueName: \"kubernetes.io/projected/c96338a8-3df3-451a-9371-124e3f270f5f-kube-api-access-w96lw\") pod \"maas-api-key-cleanup-29726535-67rs8\" (UID: \"c96338a8-3df3-451a-9371-124e3f270f5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" Jul 09 10:15:00.491646 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.491569 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" Jul 09 10:15:00.638433 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.638406 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8"] Jul 09 10:15:00.640321 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:15:00.640294 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96338a8_3df3_451a_9371_124e3f270f5f.slice/crio-43be24a74036a73cc0b70840c95c1a15a7812690d50b0b804c42bd4284d361ba WatchSource:0}: Error finding container 43be24a74036a73cc0b70840c95c1a15a7812690d50b0b804c42bd4284d361ba: Status 404 returned error can't find the container with id 43be24a74036a73cc0b70840c95c1a15a7812690d50b0b804c42bd4284d361ba Jul 09 10:15:00.642180 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.642162 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 10:15:00.835398 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:00.835361 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" event={"ID":"c96338a8-3df3-451a-9371-124e3f270f5f","Type":"ContainerStarted","Data":"43be24a74036a73cc0b70840c95c1a15a7812690d50b0b804c42bd4284d361ba"} Jul 09 10:15:03.853192 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:03.853154 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" event={"ID":"c96338a8-3df3-451a-9371-124e3f270f5f","Type":"ContainerStarted","Data":"19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4"} Jul 09 10:15:03.876475 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:15:03.876426 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" podStartSLOduration=1.149516614 podStartE2EDuration="3.876413028s" totalImagesPullingTime="2.726896414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:15:03.873559963 +0000 UTC m=+1259.206043762" watchObservedRunningTime="2026-07-09 10:15:03.876413028 +0000 UTC m=+1259.208896826" Jul 09 10:16:00.017319 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:00.017278 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8"] Jul 09 10:16:00.019609 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:00.017487 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" podUID="c96338a8-3df3-451a-9371-124e3f270f5f" containerName="cleanup" containerID="cri-o://19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4" gracePeriod=30 Jul 09 10:16:30.216247 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.216218 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" Jul 09 10:16:30.282842 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.282760 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w96lw\" (UniqueName: \"kubernetes.io/projected/c96338a8-3df3-451a-9371-124e3f270f5f-kube-api-access-w96lw\") pod \"c96338a8-3df3-451a-9371-124e3f270f5f\" (UID: \"c96338a8-3df3-451a-9371-124e3f270f5f\") " Jul 09 10:16:30.284893 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.284867 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96338a8-3df3-451a-9371-124e3f270f5f-kube-api-access-w96lw" (OuterVolumeSpecName: "kube-api-access-w96lw") pod "c96338a8-3df3-451a-9371-124e3f270f5f" (UID: "c96338a8-3df3-451a-9371-124e3f270f5f"). InnerVolumeSpecName "kube-api-access-w96lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:16:30.337327 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.337291 2571 generic.go:356] "Generic (PLEG): container finished" podID="c96338a8-3df3-451a-9371-124e3f270f5f" containerID="19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4" exitCode=137 Jul 09 10:16:30.337512 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.337344 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" Jul 09 10:16:30.337512 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.337376 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" event={"ID":"c96338a8-3df3-451a-9371-124e3f270f5f","Type":"ContainerDied","Data":"19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4"} Jul 09 10:16:30.337512 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.337427 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8" event={"ID":"c96338a8-3df3-451a-9371-124e3f270f5f","Type":"ContainerDied","Data":"43be24a74036a73cc0b70840c95c1a15a7812690d50b0b804c42bd4284d361ba"} Jul 09 10:16:30.337512 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.337449 2571 scope.go:117] "RemoveContainer" containerID="19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4" Jul 09 10:16:30.347449 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.347431 2571 scope.go:117] "RemoveContainer" containerID="19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4" Jul 09 10:16:30.347711 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:16:30.347695 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4\": container with ID starting with 19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4 not found: ID does not exist" containerID="19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4" Jul 09 10:16:30.347759 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.347721 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4"} err="failed to get container status \"19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4\": rpc error: code = NotFound desc = could not find container \"19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4\": container with ID starting with 19f42af35cf080384f46af91acd17ff16f399bed74bbd05988ff7aa2852773f4 not found: ID does not exist" Jul 09 10:16:30.362818 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.362789 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8"] Jul 09 10:16:30.369431 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.369406 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726535-67rs8"] Jul 09 10:16:30.384225 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:30.384204 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w96lw\" (UniqueName: \"kubernetes.io/projected/c96338a8-3df3-451a-9371-124e3f270f5f-kube-api-access-w96lw\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:16:31.160104 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:16:31.160074 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c96338a8-3df3-451a-9371-124e3f270f5f" path="/var/lib/kubelet/pods/c96338a8-3df3-451a-9371-124e3f270f5f/volumes" Jul 09 10:18:08.077338 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.077290 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s"] Jul 09 10:18:08.077855 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.077843 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c96338a8-3df3-451a-9371-124e3f270f5f" containerName="cleanup" Jul 09 10:18:08.077912 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.077861 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c96338a8-3df3-451a-9371-124e3f270f5f" containerName="cleanup" Jul 09 10:18:08.077948 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.077924 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c96338a8-3df3-451a-9371-124e3f270f5f" containerName="cleanup" Jul 09 10:18:08.081007 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.080990 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.083508 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.083482 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-848aa2ca-serving-cert\"" Jul 09 10:18:08.083638 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.083538 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npsbz\"" Jul 09 10:18:08.092849 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.092823 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s"] Jul 09 10:18:08.150890 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.150850 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.150890 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.150891 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jkx\" (UniqueName: \"kubernetes.io/projected/c7637e80-37e2-43be-8022-7b3a37ce5175-kube-api-access-t7jkx\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.151104 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.150979 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7637e80-37e2-43be-8022-7b3a37ce5175-openshift-service-ca\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.252250 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.252203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.252450 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:18:08.252307 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-serving-cert: secret "maas-api-e2e-ait-848aa2ca-serving-cert" not found Jul 09 10:18:08.252450 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.252374 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jkx\" (UniqueName: \"kubernetes.io/projected/c7637e80-37e2-43be-8022-7b3a37ce5175-kube-api-access-t7jkx\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.252450 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:18:08.252385 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls podName:c7637e80-37e2-43be-8022-7b3a37ce5175 nodeName:}" failed. No retries permitted until 2026-07-09 10:18:08.752360392 +0000 UTC m=+1444.084844178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls") pod "maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" (UID: "c7637e80-37e2-43be-8022-7b3a37ce5175") : secret "maas-api-e2e-ait-848aa2ca-serving-cert" not found Jul 09 10:18:08.252650 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.252478 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7637e80-37e2-43be-8022-7b3a37ce5175-openshift-service-ca\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.253110 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.253088 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7637e80-37e2-43be-8022-7b3a37ce5175-openshift-service-ca\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.264221 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.264193 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jkx\" (UniqueName: \"kubernetes.io/projected/c7637e80-37e2-43be-8022-7b3a37ce5175-kube-api-access-t7jkx\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.758200 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.758162 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.760641 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.760612 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls\") pod \"maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:08.992707 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:08.992671 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:09.149961 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:09.149939 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s"] Jul 09 10:18:09.151605 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:18:09.151579 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7637e80_37e2_43be_8022_7b3a37ce5175.slice/crio-7757c96fb4c9608c1eb7750075f73ad63b260cba107837343325cd0965c2c43e WatchSource:0}: Error finding container 7757c96fb4c9608c1eb7750075f73ad63b260cba107837343325cd0965c2c43e: Status 404 returned error can't find the container with id 7757c96fb4c9608c1eb7750075f73ad63b260cba107837343325cd0965c2c43e Jul 09 10:18:09.881649 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:09.881611 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" event={"ID":"c7637e80-37e2-43be-8022-7b3a37ce5175","Type":"ContainerStarted","Data":"7757c96fb4c9608c1eb7750075f73ad63b260cba107837343325cd0965c2c43e"} Jul 09 10:18:11.893957 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:11.893919 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" event={"ID":"c7637e80-37e2-43be-8022-7b3a37ce5175","Type":"ContainerStarted","Data":"c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988"} Jul 09 10:18:11.894433 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:11.894073 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:11.918347 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:11.918299 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" podStartSLOduration=1.7837316570000001 podStartE2EDuration="3.918277967s" totalImagesPullingTime="2.13454631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:18:11.91801296 +0000 UTC m=+1447.250496759" watchObservedRunningTime="2026-07-09 10:18:11.918277967 +0000 UTC m=+1447.250761766" Jul 09 10:18:13.768440 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:13.768317 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s"] Jul 09 10:18:13.905437 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:13.905391 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" podUID="c7637e80-37e2-43be-8022-7b3a37ce5175" containerName="maas-api" containerID="cri-o://c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988" gracePeriod=30 Jul 09 10:18:14.158813 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.158786 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:14.208356 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.208321 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7637e80-37e2-43be-8022-7b3a37ce5175-openshift-service-ca\") pod \"c7637e80-37e2-43be-8022-7b3a37ce5175\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " Jul 09 10:18:14.208555 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.208368 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls\") pod \"c7637e80-37e2-43be-8022-7b3a37ce5175\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " Jul 09 10:18:14.208555 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.208408 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7jkx\" (UniqueName: \"kubernetes.io/projected/c7637e80-37e2-43be-8022-7b3a37ce5175-kube-api-access-t7jkx\") pod \"c7637e80-37e2-43be-8022-7b3a37ce5175\" (UID: \"c7637e80-37e2-43be-8022-7b3a37ce5175\") " Jul 09 10:18:14.208726 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.208691 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7637e80-37e2-43be-8022-7b3a37ce5175-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7637e80-37e2-43be-8022-7b3a37ce5175" (UID: "c7637e80-37e2-43be-8022-7b3a37ce5175"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:18:14.210641 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.210608 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7637e80-37e2-43be-8022-7b3a37ce5175" (UID: "c7637e80-37e2-43be-8022-7b3a37ce5175"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:18:14.210772 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.210688 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7637e80-37e2-43be-8022-7b3a37ce5175-kube-api-access-t7jkx" (OuterVolumeSpecName: "kube-api-access-t7jkx") pod "c7637e80-37e2-43be-8022-7b3a37ce5175" (UID: "c7637e80-37e2-43be-8022-7b3a37ce5175"). InnerVolumeSpecName "kube-api-access-t7jkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:18:14.309279 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.309199 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7jkx\" (UniqueName: \"kubernetes.io/projected/c7637e80-37e2-43be-8022-7b3a37ce5175-kube-api-access-t7jkx\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:14.309279 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.309226 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7637e80-37e2-43be-8022-7b3a37ce5175-openshift-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:14.309279 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.309235 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7637e80-37e2-43be-8022-7b3a37ce5175-maas-api-tls\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:14.912251 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.912212 2571 generic.go:356] "Generic (PLEG): container finished" podID="c7637e80-37e2-43be-8022-7b3a37ce5175" containerID="c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988" exitCode=0 Jul 09 10:18:14.912704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.912263 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" Jul 09 10:18:14.912704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.912294 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" event={"ID":"c7637e80-37e2-43be-8022-7b3a37ce5175","Type":"ContainerDied","Data":"c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988"} Jul 09 10:18:14.912704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.912345 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s" event={"ID":"c7637e80-37e2-43be-8022-7b3a37ce5175","Type":"ContainerDied","Data":"7757c96fb4c9608c1eb7750075f73ad63b260cba107837343325cd0965c2c43e"} Jul 09 10:18:14.912704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.912367 2571 scope.go:117] "RemoveContainer" containerID="c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988" Jul 09 10:18:14.922483 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.922463 2571 scope.go:117] "RemoveContainer" containerID="c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988" Jul 09 10:18:14.922773 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:18:14.922744 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988\": container with ID starting with c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988 not found: ID does not exist" containerID="c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988" Jul 09 10:18:14.922891 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.922779 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988"} err="failed to get container status \"c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988\": rpc error: code = NotFound desc = could not find container \"c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988\": container with ID starting with c32f9995918d56ab6c7f04a97883ffcc8de77306d5d89a7440224f42b83ad988 not found: ID does not exist" Jul 09 10:18:14.936940 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.936913 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s"] Jul 09 10:18:14.942945 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:14.942923 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-848aa2ca-6f556b9695-nh54s"] Jul 09 10:18:15.161629 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:15.161548 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7637e80-37e2-43be-8022-7b3a37ce5175" path="/var/lib/kubelet/pods/c7637e80-37e2-43be-8022-7b3a37ce5175/volumes" Jul 09 10:18:21.449111 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.449072 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm"] Jul 09 10:18:21.449717 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.449698 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7637e80-37e2-43be-8022-7b3a37ce5175" containerName="maas-api" Jul 09 10:18:21.449779 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.449721 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7637e80-37e2-43be-8022-7b3a37ce5175" containerName="maas-api" Jul 09 10:18:21.449853 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.449842 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7637e80-37e2-43be-8022-7b3a37ce5175" containerName="maas-api" Jul 09 10:18:21.454671 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.454648 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:21.457841 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.457666 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ba8169ec-serving-cert\"" Jul 09 10:18:21.458788 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.458762 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npsbz\"" Jul 09 10:18:21.462879 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.462852 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm"] Jul 09 10:18:21.580372 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.580334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ckr\" (UniqueName: \"kubernetes.io/projected/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-kube-api-access-f5ckr\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:21.580534 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.580375 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:21.580534 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.580443 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-openshift-service-ca\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:21.681959 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.681916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ckr\" (UniqueName: \"kubernetes.io/projected/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-kube-api-access-f5ckr\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:21.681959 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.681964 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:21.682239 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.682015 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-openshift-service-ca\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:21.682239 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:18:21.682120 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-serving-cert: secret "maas-api-e2e-ait-ba8169ec-serving-cert" not found Jul 09 10:18:21.682239 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:18:21.682210 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls podName:e579cdb9-629d-4bfd-84ab-36ac7a5d8353 nodeName:}" failed. No retries permitted until 2026-07-09 10:18:22.182194874 +0000 UTC m=+1457.514678650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls") pod "maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" (UID: "e579cdb9-629d-4bfd-84ab-36ac7a5d8353") : secret "maas-api-e2e-ait-ba8169ec-serving-cert" not found Jul 09 10:18:21.682740 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.682718 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-openshift-service-ca\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:21.694489 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:21.694464 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ckr\" (UniqueName: \"kubernetes.io/projected/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-kube-api-access-f5ckr\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:22.186585 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:22.186549 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:22.189283 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:22.189252 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls\") pod \"maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:22.368594 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:22.368556 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:22.512467 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:22.512430 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm"] Jul 09 10:18:22.514404 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:18:22.514374 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode579cdb9_629d_4bfd_84ab_36ac7a5d8353.slice/crio-20d59061a9adda2f0ac2630f0e4f91a6f9f6e11dafe302de48792b21c5a6f192 WatchSource:0}: Error finding container 20d59061a9adda2f0ac2630f0e4f91a6f9f6e11dafe302de48792b21c5a6f192: Status 404 returned error can't find the container with id 20d59061a9adda2f0ac2630f0e4f91a6f9f6e11dafe302de48792b21c5a6f192 Jul 09 10:18:22.968725 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:22.968687 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" event={"ID":"e579cdb9-629d-4bfd-84ab-36ac7a5d8353","Type":"ContainerStarted","Data":"20d59061a9adda2f0ac2630f0e4f91a6f9f6e11dafe302de48792b21c5a6f192"} Jul 09 10:18:23.975703 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:23.975666 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" event={"ID":"e579cdb9-629d-4bfd-84ab-36ac7a5d8353","Type":"ContainerStarted","Data":"cb5f46ff77b67c6e644f5dc1e112000d2df3abeffd5abae6090044234b409694"} Jul 09 10:18:23.976201 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:23.975738 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:23.995113 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:23.995061 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" podStartSLOduration=2.492932764 podStartE2EDuration="2.995045527s" totalImagesPullingTime="502.112763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:18:23.99395342 +0000 UTC m=+1459.326437218" watchObservedRunningTime="2026-07-09 10:18:23.995045527 +0000 UTC m=+1459.327529375" Jul 09 10:18:26.865163 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:26.865094 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm"] Jul 09 10:18:26.865699 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:26.865319 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" podUID="e579cdb9-629d-4bfd-84ab-36ac7a5d8353" containerName="maas-api" containerID="cri-o://cb5f46ff77b67c6e644f5dc1e112000d2df3abeffd5abae6090044234b409694" gracePeriod=30 Jul 09 10:18:27.001142 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.001093 2571 generic.go:356] "Generic (PLEG): container finished" podID="e579cdb9-629d-4bfd-84ab-36ac7a5d8353" containerID="cb5f46ff77b67c6e644f5dc1e112000d2df3abeffd5abae6090044234b409694" exitCode=0 Jul 09 10:18:27.001313 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.001166 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" event={"ID":"e579cdb9-629d-4bfd-84ab-36ac7a5d8353","Type":"ContainerDied","Data":"cb5f46ff77b67c6e644f5dc1e112000d2df3abeffd5abae6090044234b409694"} Jul 09 10:18:27.135709 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.135679 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:27.236183 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.236155 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5ckr\" (UniqueName: \"kubernetes.io/projected/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-kube-api-access-f5ckr\") pod \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " Jul 09 10:18:27.236320 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.236227 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-openshift-service-ca\") pod \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " Jul 09 10:18:27.236320 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.236264 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls\") pod \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\" (UID: \"e579cdb9-629d-4bfd-84ab-36ac7a5d8353\") " Jul 09 10:18:27.236613 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.236582 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e579cdb9-629d-4bfd-84ab-36ac7a5d8353" (UID: "e579cdb9-629d-4bfd-84ab-36ac7a5d8353"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:18:27.238330 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.238310 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e579cdb9-629d-4bfd-84ab-36ac7a5d8353" (UID: "e579cdb9-629d-4bfd-84ab-36ac7a5d8353"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:18:27.238425 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.238405 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-kube-api-access-f5ckr" (OuterVolumeSpecName: "kube-api-access-f5ckr") pod "e579cdb9-629d-4bfd-84ab-36ac7a5d8353" (UID: "e579cdb9-629d-4bfd-84ab-36ac7a5d8353"). InnerVolumeSpecName "kube-api-access-f5ckr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:18:27.337179 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.337119 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5ckr\" (UniqueName: \"kubernetes.io/projected/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-kube-api-access-f5ckr\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:27.337179 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.337176 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-openshift-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:27.337179 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:27.337186 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e579cdb9-629d-4bfd-84ab-36ac7a5d8353-maas-api-tls\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:28.008377 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:28.008346 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" Jul 09 10:18:28.008785 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:28.008350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm" event={"ID":"e579cdb9-629d-4bfd-84ab-36ac7a5d8353","Type":"ContainerDied","Data":"20d59061a9adda2f0ac2630f0e4f91a6f9f6e11dafe302de48792b21c5a6f192"} Jul 09 10:18:28.008785 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:28.008478 2571 scope.go:117] "RemoveContainer" containerID="cb5f46ff77b67c6e644f5dc1e112000d2df3abeffd5abae6090044234b409694" Jul 09 10:18:28.032192 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:28.032164 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm"] Jul 09 10:18:28.036157 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:28.036137 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ba8169ec-7dd57b86cd-5d5lm"] Jul 09 10:18:29.161874 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:29.161838 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e579cdb9-629d-4bfd-84ab-36ac7a5d8353" path="/var/lib/kubelet/pods/e579cdb9-629d-4bfd-84ab-36ac7a5d8353/volumes" Jul 09 10:18:41.410357 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.410313 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl"] Jul 09 10:18:41.410785 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.410748 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e579cdb9-629d-4bfd-84ab-36ac7a5d8353" containerName="maas-api" Jul 09 10:18:41.410785 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.410764 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e579cdb9-629d-4bfd-84ab-36ac7a5d8353" containerName="maas-api" Jul 09 10:18:41.410858 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.410852 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e579cdb9-629d-4bfd-84ab-36ac7a5d8353" containerName="maas-api" Jul 09 10:18:41.413997 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.413974 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.417376 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.417350 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-92153720-serving-cert\"" Jul 09 10:18:41.417564 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.417367 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npsbz\"" Jul 09 10:18:41.431830 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.431793 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl"] Jul 09 10:18:41.462923 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.462882 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4308fcfe-7de7-4385-ab43-1522353c0ff3-openshift-service-ca\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.463100 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.462931 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4308fcfe-7de7-4385-ab43-1522353c0ff3-maas-api-tls\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.463100 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.462957 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbgt\" (UniqueName: \"kubernetes.io/projected/4308fcfe-7de7-4385-ab43-1522353c0ff3-kube-api-access-vfbgt\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.564086 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.564051 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4308fcfe-7de7-4385-ab43-1522353c0ff3-openshift-service-ca\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.564285 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.564099 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4308fcfe-7de7-4385-ab43-1522353c0ff3-maas-api-tls\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.564285 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.564252 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbgt\" (UniqueName: \"kubernetes.io/projected/4308fcfe-7de7-4385-ab43-1522353c0ff3-kube-api-access-vfbgt\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.564891 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.564865 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4308fcfe-7de7-4385-ab43-1522353c0ff3-openshift-service-ca\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.566522 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.566503 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4308fcfe-7de7-4385-ab43-1522353c0ff3-maas-api-tls\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.574558 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.574529 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbgt\" (UniqueName: \"kubernetes.io/projected/4308fcfe-7de7-4385-ab43-1522353c0ff3-kube-api-access-vfbgt\") pod \"maas-api-e2e-derive-92153720-694f594769-4k9bl\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.730405 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.730309 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:41.887470 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:41.887443 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl"] Jul 09 10:18:41.889338 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:18:41.889310 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4308fcfe_7de7_4385_ab43_1522353c0ff3.slice/crio-b8e1f6e401993e782e877a805d8e55b95eca777b984e7a00faf5b200c2fe0c36 WatchSource:0}: Error finding container b8e1f6e401993e782e877a805d8e55b95eca777b984e7a00faf5b200c2fe0c36: Status 404 returned error can't find the container with id b8e1f6e401993e782e877a805d8e55b95eca777b984e7a00faf5b200c2fe0c36 Jul 09 10:18:42.087255 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:42.087222 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" event={"ID":"4308fcfe-7de7-4385-ab43-1522353c0ff3","Type":"ContainerStarted","Data":"b8e1f6e401993e782e877a805d8e55b95eca777b984e7a00faf5b200c2fe0c36"} Jul 09 10:18:43.094325 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:43.094286 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" event={"ID":"4308fcfe-7de7-4385-ab43-1522353c0ff3","Type":"ContainerStarted","Data":"02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e"} Jul 09 10:18:43.094743 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:43.094351 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:43.117258 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:43.117193 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" podStartSLOduration=1.738727442 podStartE2EDuration="2.117176315s" totalImagesPullingTime="378.448873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:18:43.115257476 +0000 UTC m=+1478.447741288" watchObservedRunningTime="2026-07-09 10:18:43.117176315 +0000 UTC m=+1478.449660112" Jul 09 10:18:46.058771 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.058713 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl"] Jul 09 10:18:46.059856 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.059822 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" podUID="4308fcfe-7de7-4385-ab43-1522353c0ff3" containerName="maas-api" containerID="cri-o://02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e" gracePeriod=30 Jul 09 10:18:46.316612 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.316589 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:46.408208 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.408181 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4308fcfe-7de7-4385-ab43-1522353c0ff3-openshift-service-ca\") pod \"4308fcfe-7de7-4385-ab43-1522353c0ff3\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " Jul 09 10:18:46.408388 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.408239 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4308fcfe-7de7-4385-ab43-1522353c0ff3-maas-api-tls\") pod \"4308fcfe-7de7-4385-ab43-1522353c0ff3\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " Jul 09 10:18:46.408388 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.408283 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfbgt\" (UniqueName: \"kubernetes.io/projected/4308fcfe-7de7-4385-ab43-1522353c0ff3-kube-api-access-vfbgt\") pod \"4308fcfe-7de7-4385-ab43-1522353c0ff3\" (UID: \"4308fcfe-7de7-4385-ab43-1522353c0ff3\") " Jul 09 10:18:46.408571 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.408546 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4308fcfe-7de7-4385-ab43-1522353c0ff3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4308fcfe-7de7-4385-ab43-1522353c0ff3" (UID: "4308fcfe-7de7-4385-ab43-1522353c0ff3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:18:46.410392 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.410360 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4308fcfe-7de7-4385-ab43-1522353c0ff3-kube-api-access-vfbgt" (OuterVolumeSpecName: "kube-api-access-vfbgt") pod "4308fcfe-7de7-4385-ab43-1522353c0ff3" (UID: "4308fcfe-7de7-4385-ab43-1522353c0ff3"). InnerVolumeSpecName "kube-api-access-vfbgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:18:46.410496 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.410455 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4308fcfe-7de7-4385-ab43-1522353c0ff3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4308fcfe-7de7-4385-ab43-1522353c0ff3" (UID: "4308fcfe-7de7-4385-ab43-1522353c0ff3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:18:46.509199 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.509171 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4308fcfe-7de7-4385-ab43-1522353c0ff3-maas-api-tls\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:46.509199 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.509201 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfbgt\" (UniqueName: \"kubernetes.io/projected/4308fcfe-7de7-4385-ab43-1522353c0ff3-kube-api-access-vfbgt\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:46.509402 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:46.509217 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4308fcfe-7de7-4385-ab43-1522353c0ff3-openshift-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:18:47.116897 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.116861 2571 generic.go:356] "Generic (PLEG): container finished" podID="4308fcfe-7de7-4385-ab43-1522353c0ff3" containerID="02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e" exitCode=0 Jul 09 10:18:47.117409 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.116908 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" Jul 09 10:18:47.117409 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.116911 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" event={"ID":"4308fcfe-7de7-4385-ab43-1522353c0ff3","Type":"ContainerDied","Data":"02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e"} Jul 09 10:18:47.117409 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.116963 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl" event={"ID":"4308fcfe-7de7-4385-ab43-1522353c0ff3","Type":"ContainerDied","Data":"b8e1f6e401993e782e877a805d8e55b95eca777b984e7a00faf5b200c2fe0c36"} Jul 09 10:18:47.117409 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.116992 2571 scope.go:117] "RemoveContainer" containerID="02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e" Jul 09 10:18:47.127634 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.127610 2571 scope.go:117] "RemoveContainer" containerID="02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e" Jul 09 10:18:47.127953 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:18:47.127924 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e\": container with ID starting with 02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e not found: ID does not exist" containerID="02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e" Jul 09 10:18:47.128045 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.127964 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e"} err="failed to get container status \"02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e\": rpc error: code = NotFound desc = could not find container \"02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e\": container with ID starting with 02827d48f8ec25a2e59617ce2265506ceb57f8a6e7cf39011e7f65dfc7a99f7e not found: ID does not exist" Jul 09 10:18:47.143748 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.143720 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl"] Jul 09 10:18:47.148479 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.148458 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-92153720-694f594769-4k9bl"] Jul 09 10:18:47.160236 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:18:47.160215 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4308fcfe-7de7-4385-ab43-1522353c0ff3" path="/var/lib/kubelet/pods/4308fcfe-7de7-4385-ab43-1522353c0ff3/volumes" Jul 09 10:19:05.194670 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:19:05.194644 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:19:05.197998 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:19:05.197973 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:20:37.031307 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.031269 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr"] Jul 09 10:20:37.031818 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.031706 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4308fcfe-7de7-4385-ab43-1522353c0ff3" containerName="maas-api" Jul 09 10:20:37.031818 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.031722 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4308fcfe-7de7-4385-ab43-1522353c0ff3" containerName="maas-api" Jul 09 10:20:37.031818 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.031789 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4308fcfe-7de7-4385-ab43-1522353c0ff3" containerName="maas-api" Jul 09 10:20:37.034946 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.034929 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.040108 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.040087 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1df781d2-serving-cert\"" Jul 09 10:20:37.040274 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.040256 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npsbz\"" Jul 09 10:20:37.082502 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.082465 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-openshift-service-ca\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.082502 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.082504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rxhk\" (UniqueName: \"kubernetes.io/projected/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-kube-api-access-9rxhk\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.082715 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.082545 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-maas-api-tls\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.091268 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.091237 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr"] Jul 09 10:20:37.183665 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.183626 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-openshift-service-ca\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.183665 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.183670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxhk\" (UniqueName: \"kubernetes.io/projected/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-kube-api-access-9rxhk\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.183883 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.183716 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-maas-api-tls\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.184359 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.184332 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-openshift-service-ca\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.186161 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.186105 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-maas-api-tls\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.204749 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.204721 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxhk\" (UniqueName: \"kubernetes.io/projected/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-kube-api-access-9rxhk\") pod \"maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.345805 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.345771 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:37.497038 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.497005 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr"] Jul 09 10:20:37.498011 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:20:37.497975 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3bedf2_a4b0_45cc_9897_60b8686ffd07.slice/crio-8f1d3ac6f0c984c0ba8888593568844816a2cb4de360eb7ffe70feefb860e785 WatchSource:0}: Error finding container 8f1d3ac6f0c984c0ba8888593568844816a2cb4de360eb7ffe70feefb860e785: Status 404 returned error can't find the container with id 8f1d3ac6f0c984c0ba8888593568844816a2cb4de360eb7ffe70feefb860e785 Jul 09 10:20:37.499691 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.499673 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 10:20:37.718105 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:37.718020 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" event={"ID":"fe3bedf2-a4b0-45cc-9897-60b8686ffd07","Type":"ContainerStarted","Data":"8f1d3ac6f0c984c0ba8888593568844816a2cb4de360eb7ffe70feefb860e785"} Jul 09 10:20:38.728235 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:38.728195 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" event={"ID":"fe3bedf2-a4b0-45cc-9897-60b8686ffd07","Type":"ContainerStarted","Data":"601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af"} Jul 09 10:20:38.729255 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:38.729226 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:38.748800 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:38.748738 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" podStartSLOduration=1.946238812 podStartE2EDuration="2.748716013s" totalImagesPullingTime="802.477201ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:20:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:20:38.748301639 +0000 UTC m=+1594.080785438" watchObservedRunningTime="2026-07-09 10:20:38.748716013 +0000 UTC m=+1594.081199812" Jul 09 10:20:45.746800 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:45.746767 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:50.000981 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.000935 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr"] Jul 09 10:20:50.001426 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.001184 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" podUID="fe3bedf2-a4b0-45cc-9897-60b8686ffd07" containerName="maas-api" containerID="cri-o://601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af" gracePeriod=30 Jul 09 10:20:50.250107 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.250077 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:50.408111 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.408075 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-maas-api-tls\") pod \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " Jul 09 10:20:50.408381 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.408186 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-openshift-service-ca\") pod \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " Jul 09 10:20:50.408381 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.408207 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rxhk\" (UniqueName: \"kubernetes.io/projected/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-kube-api-access-9rxhk\") pod \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\" (UID: \"fe3bedf2-a4b0-45cc-9897-60b8686ffd07\") " Jul 09 10:20:50.408598 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.408573 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe3bedf2-a4b0-45cc-9897-60b8686ffd07" (UID: "fe3bedf2-a4b0-45cc-9897-60b8686ffd07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:20:50.410187 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.410162 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe3bedf2-a4b0-45cc-9897-60b8686ffd07" (UID: "fe3bedf2-a4b0-45cc-9897-60b8686ffd07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:20:50.410247 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.410171 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-kube-api-access-9rxhk" (OuterVolumeSpecName: "kube-api-access-9rxhk") pod "fe3bedf2-a4b0-45cc-9897-60b8686ffd07" (UID: "fe3bedf2-a4b0-45cc-9897-60b8686ffd07"). InnerVolumeSpecName "kube-api-access-9rxhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:20:50.509608 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.509573 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-openshift-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:20:50.509608 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.509602 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rxhk\" (UniqueName: \"kubernetes.io/projected/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-kube-api-access-9rxhk\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:20:50.509608 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.509612 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe3bedf2-a4b0-45cc-9897-60b8686ffd07-maas-api-tls\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:20:50.798347 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.798252 2571 generic.go:356] "Generic (PLEG): container finished" podID="fe3bedf2-a4b0-45cc-9897-60b8686ffd07" containerID="601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af" exitCode=0 Jul 09 10:20:50.798347 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.798303 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" Jul 09 10:20:50.798548 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.798339 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" event={"ID":"fe3bedf2-a4b0-45cc-9897-60b8686ffd07","Type":"ContainerDied","Data":"601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af"} Jul 09 10:20:50.798548 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.798398 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr" event={"ID":"fe3bedf2-a4b0-45cc-9897-60b8686ffd07","Type":"ContainerDied","Data":"8f1d3ac6f0c984c0ba8888593568844816a2cb4de360eb7ffe70feefb860e785"} Jul 09 10:20:50.798548 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.798420 2571 scope.go:117] "RemoveContainer" containerID="601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af" Jul 09 10:20:50.808965 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.808945 2571 scope.go:117] "RemoveContainer" containerID="601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af" Jul 09 10:20:50.809276 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:20:50.809255 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af\": container with ID starting with 601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af not found: ID does not exist" containerID="601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af" Jul 09 10:20:50.809361 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.809281 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af"} err="failed to get container status \"601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af\": rpc error: code = NotFound desc = could not find container \"601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af\": container with ID starting with 601a68c81f486b76c029467e23ec22390f7c824cdfe4690dc8976604780884af not found: ID does not exist" Jul 09 10:20:50.822774 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.822736 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr"] Jul 09 10:20:50.827593 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:50.827565 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1df781d2-5b96b47b5c-j25sr"] Jul 09 10:20:51.167112 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:20:51.167070 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe3bedf2-a4b0-45cc-9897-60b8686ffd07" path="/var/lib/kubelet/pods/fe3bedf2-a4b0-45cc-9897-60b8686ffd07/volumes" Jul 09 10:21:21.185850 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.185812 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv"] Jul 09 10:21:21.186323 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.186265 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe3bedf2-a4b0-45cc-9897-60b8686ffd07" containerName="maas-api" Jul 09 10:21:21.186323 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.186277 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe3bedf2-a4b0-45cc-9897-60b8686ffd07" containerName="maas-api" Jul 09 10:21:21.186398 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.186347 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe3bedf2-a4b0-45cc-9897-60b8686ffd07" containerName="maas-api" Jul 09 10:21:21.189525 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.189506 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.192421 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.192360 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npsbz\"" Jul 09 10:21:21.192968 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.192953 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7dafaaa7-serving-cert\"" Jul 09 10:21:21.199915 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.199894 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv"] Jul 09 10:21:21.298188 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.298110 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-openshift-service-ca\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.298379 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.298244 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwdb\" (UniqueName: \"kubernetes.io/projected/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-kube-api-access-gmwdb\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.298379 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.298290 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.399185 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.399115 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-openshift-service-ca\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.399391 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.399213 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwdb\" (UniqueName: \"kubernetes.io/projected/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-kube-api-access-gmwdb\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.399391 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.399260 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.399494 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:21:21.399449 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-serving-cert: secret "maas-api-e2e-mt-7dafaaa7-serving-cert" not found Jul 09 10:21:21.399535 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:21:21.399529 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls podName:b7b79da6-e6b6-4bfb-be00-ba0debdabb42 nodeName:}" failed. No retries permitted until 2026-07-09 10:21:21.899505097 +0000 UTC m=+1637.231988886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls") pod "maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" (UID: "b7b79da6-e6b6-4bfb-be00-ba0debdabb42") : secret "maas-api-e2e-mt-7dafaaa7-serving-cert" not found Jul 09 10:21:21.399905 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.399879 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-openshift-service-ca\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.412371 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.412344 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwdb\" (UniqueName: \"kubernetes.io/projected/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-kube-api-access-gmwdb\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.905433 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.905389 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:21.907976 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:21.907944 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls\") pod \"maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:22.101766 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:22.101733 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:22.246298 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:22.246273 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv"] Jul 09 10:21:22.249685 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:21:22.249656 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b79da6_e6b6_4bfb_be00_ba0debdabb42.slice/crio-1480f89a99b7f3cf60f798334e258ba40caf575c2df4ba42b50b8f1d19471171 WatchSource:0}: Error finding container 1480f89a99b7f3cf60f798334e258ba40caf575c2df4ba42b50b8f1d19471171: Status 404 returned error can't find the container with id 1480f89a99b7f3cf60f798334e258ba40caf575c2df4ba42b50b8f1d19471171 Jul 09 10:21:22.983236 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:22.983146 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" event={"ID":"b7b79da6-e6b6-4bfb-be00-ba0debdabb42","Type":"ContainerStarted","Data":"4faded8da119f8e9831cd5b09fc175078a2a2dfd7af630390b464e07e0bbadca"} Jul 09 10:21:22.983236 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:22.983200 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" event={"ID":"b7b79da6-e6b6-4bfb-be00-ba0debdabb42","Type":"ContainerStarted","Data":"1480f89a99b7f3cf60f798334e258ba40caf575c2df4ba42b50b8f1d19471171"} Jul 09 10:21:22.983435 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:22.983243 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:23.004475 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:23.004427 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" podStartSLOduration=1.657230349 podStartE2EDuration="2.004409965s" totalImagesPullingTime="347.179616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:21:23.001688071 +0000 UTC m=+1638.334171869" watchObservedRunningTime="2026-07-09 10:21:23.004409965 +0000 UTC m=+1638.336893763" Jul 09 10:21:28.994818 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:28.994788 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:58.971422 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:58.971388 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv"] Jul 09 10:21:58.972025 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:58.971675 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" podUID="b7b79da6-e6b6-4bfb-be00-ba0debdabb42" containerName="maas-api" containerID="cri-o://4faded8da119f8e9831cd5b09fc175078a2a2dfd7af630390b464e07e0bbadca" gracePeriod=30 Jul 09 10:21:58.989774 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:58.989744 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" podUID="b7b79da6-e6b6-4bfb-be00-ba0debdabb42" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.69:8443/health\": dial tcp 10.132.0.69:8443: connect: connection refused" Jul 09 10:21:59.185622 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.185587 2571 generic.go:356] "Generic (PLEG): container finished" podID="b7b79da6-e6b6-4bfb-be00-ba0debdabb42" containerID="4faded8da119f8e9831cd5b09fc175078a2a2dfd7af630390b464e07e0bbadca" exitCode=0 Jul 09 10:21:59.185799 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.185666 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" event={"ID":"b7b79da6-e6b6-4bfb-be00-ba0debdabb42","Type":"ContainerDied","Data":"4faded8da119f8e9831cd5b09fc175078a2a2dfd7af630390b464e07e0bbadca"} Jul 09 10:21:59.228499 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.228440 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:21:59.342830 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.342799 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls\") pod \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " Jul 09 10:21:59.343029 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.342843 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmwdb\" (UniqueName: \"kubernetes.io/projected/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-kube-api-access-gmwdb\") pod \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " Jul 09 10:21:59.343029 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.342911 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-openshift-service-ca\") pod \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\" (UID: \"b7b79da6-e6b6-4bfb-be00-ba0debdabb42\") " Jul 09 10:21:59.343351 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.343324 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7b79da6-e6b6-4bfb-be00-ba0debdabb42" (UID: "b7b79da6-e6b6-4bfb-be00-ba0debdabb42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:21:59.344897 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.344876 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7b79da6-e6b6-4bfb-be00-ba0debdabb42" (UID: "b7b79da6-e6b6-4bfb-be00-ba0debdabb42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:21:59.345002 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.344983 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-kube-api-access-gmwdb" (OuterVolumeSpecName: "kube-api-access-gmwdb") pod "b7b79da6-e6b6-4bfb-be00-ba0debdabb42" (UID: "b7b79da6-e6b6-4bfb-be00-ba0debdabb42"). InnerVolumeSpecName "kube-api-access-gmwdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:21:59.444349 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.444314 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-maas-api-tls\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:21:59.444349 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.444343 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmwdb\" (UniqueName: \"kubernetes.io/projected/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-kube-api-access-gmwdb\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:21:59.444349 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:21:59.444354 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7b79da6-e6b6-4bfb-be00-ba0debdabb42-openshift-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:22:00.191909 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:22:00.191878 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" Jul 09 10:22:00.192338 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:22:00.191903 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv" event={"ID":"b7b79da6-e6b6-4bfb-be00-ba0debdabb42","Type":"ContainerDied","Data":"1480f89a99b7f3cf60f798334e258ba40caf575c2df4ba42b50b8f1d19471171"} Jul 09 10:22:00.192338 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:22:00.191969 2571 scope.go:117] "RemoveContainer" containerID="4faded8da119f8e9831cd5b09fc175078a2a2dfd7af630390b464e07e0bbadca" Jul 09 10:22:00.216317 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:22:00.216292 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv"] Jul 09 10:22:00.222059 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:22:00.222032 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7dafaaa7-685bcd894-kpbcv"] Jul 09 10:22:01.160319 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:22:01.160283 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7b79da6-e6b6-4bfb-be00-ba0debdabb42" path="/var/lib/kubelet/pods/b7b79da6-e6b6-4bfb-be00-ba0debdabb42/volumes" Jul 09 10:23:57.019894 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.019856 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp"] Jul 09 10:23:57.020499 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.020481 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b79da6-e6b6-4bfb-be00-ba0debdabb42" containerName="maas-api" Jul 09 10:23:57.020547 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.020502 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b79da6-e6b6-4bfb-be00-ba0debdabb42" containerName="maas-api" Jul 09 10:23:57.020610 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.020599 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b79da6-e6b6-4bfb-be00-ba0debdabb42" containerName="maas-api" Jul 09 10:23:57.024159 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.024120 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.027299 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.027274 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npsbz\"" Jul 09 10:23:57.027472 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.027274 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-31eb62-serving-cert\"" Jul 09 10:23:57.034061 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.034041 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp"] Jul 09 10:23:57.193992 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.193913 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.193992 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.193950 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkmz\" (UniqueName: \"kubernetes.io/projected/770a88d4-2ff4-4ecc-bcfa-f61229894172-kube-api-access-9dkmz\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.193992 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.193982 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a88d4-2ff4-4ecc-bcfa-f61229894172-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.295307 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.295226 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.295307 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.295266 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkmz\" (UniqueName: \"kubernetes.io/projected/770a88d4-2ff4-4ecc-bcfa-f61229894172-kube-api-access-9dkmz\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.295307 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.295302 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a88d4-2ff4-4ecc-bcfa-f61229894172-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.295596 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:23:57.295393 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-serving-cert: secret "maas-api-e2e-inf-a-31eb62-serving-cert" not found Jul 09 10:23:57.295596 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:23:57.295464 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls podName:770a88d4-2ff4-4ecc-bcfa-f61229894172 nodeName:}" failed. No retries permitted until 2026-07-09 10:23:57.795449741 +0000 UTC m=+1793.127933518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls") pod "maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" (UID: "770a88d4-2ff4-4ecc-bcfa-f61229894172") : secret "maas-api-e2e-inf-a-31eb62-serving-cert" not found Jul 09 10:23:57.295926 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.295907 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a88d4-2ff4-4ecc-bcfa-f61229894172-openshift-service-ca\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.310812 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.310787 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkmz\" (UniqueName: \"kubernetes.io/projected/770a88d4-2ff4-4ecc-bcfa-f61229894172-kube-api-access-9dkmz\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.800451 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.800415 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.802841 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.802818 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls\") pod \"maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:57.937749 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:57.937711 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:58.112864 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:58.112830 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp"] Jul 09 10:23:58.114471 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:23:58.114443 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770a88d4_2ff4_4ecc_bcfa_f61229894172.slice/crio-e5d620681cd68dadb008d7dc831fb4d20e631c8be211908ef8cd4bbd2eddde1d WatchSource:0}: Error finding container e5d620681cd68dadb008d7dc831fb4d20e631c8be211908ef8cd4bbd2eddde1d: Status 404 returned error can't find the container with id e5d620681cd68dadb008d7dc831fb4d20e631c8be211908ef8cd4bbd2eddde1d Jul 09 10:23:58.847769 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:58.847679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" event={"ID":"770a88d4-2ff4-4ecc-bcfa-f61229894172","Type":"ContainerStarted","Data":"7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6"} Jul 09 10:23:58.847769 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:58.847743 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:23:58.847769 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:58.847766 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" event={"ID":"770a88d4-2ff4-4ecc-bcfa-f61229894172","Type":"ContainerStarted","Data":"e5d620681cd68dadb008d7dc831fb4d20e631c8be211908ef8cd4bbd2eddde1d"} Jul 09 10:23:58.869606 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:23:58.869559 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" podStartSLOduration=2.410680912 podStartE2EDuration="2.869544854s" totalImagesPullingTime="458.863942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:23:58.868887085 +0000 UTC m=+1794.201370885" watchObservedRunningTime="2026-07-09 10:23:58.869544854 +0000 UTC m=+1794.202028651" Jul 09 10:24:02.233258 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.233220 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5"] Jul 09 10:24:02.237599 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.237578 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.240365 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.240341 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-14869d-openshift-default-dockercfg-s6rhb\"" Jul 09 10:24:02.240471 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.240402 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 10:24:02.250301 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.250279 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5"] Jul 09 10:24:02.345521 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ef772-b5ae-413e-bed5-18b88335d224-istiod-ca-cert\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.345699 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345532 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-data\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.345699 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345565 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r66cr\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-kube-api-access-r66cr\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.345699 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345593 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-credential-socket\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.345699 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345648 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-podinfo\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.345699 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345693 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-envoy\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.345900 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345740 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-token\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.345900 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345767 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-socket\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.345900 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.345810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-certs\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447118 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-token\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447341 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-socket\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447341 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447217 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-certs\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447341 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447318 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ef772-b5ae-413e-bed5-18b88335d224-istiod-ca-cert\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447522 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447364 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-data\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447522 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447392 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r66cr\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-kube-api-access-r66cr\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447522 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-credential-socket\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447522 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447471 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-podinfo\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447522 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447502 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-envoy\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447775 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447636 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-certs\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447775 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447733 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-socket\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447887 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447837 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-data\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.447887 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.447880 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-credential-socket\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.448107 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.448089 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ef772-b5ae-413e-bed5-18b88335d224-istiod-ca-cert\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.449729 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.449706 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-envoy\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.449958 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.449939 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-podinfo\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.456828 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.456800 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-token\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.457218 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.457202 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r66cr\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-kube-api-access-r66cr\") pod \"e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.547882 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.547797 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:02.698056 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.698024 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5"] Jul 09 10:24:02.701510 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:24:02.701323 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6ef772_b5ae_413e_bed5_18b88335d224.slice/crio-07f67299a70214a44a934030386a4ff4c38d5f93675d9493508c361dfae88375 WatchSource:0}: Error finding container 07f67299a70214a44a934030386a4ff4c38d5f93675d9493508c361dfae88375: Status 404 returned error can't find the container with id 07f67299a70214a44a934030386a4ff4c38d5f93675d9493508c361dfae88375 Jul 09 10:24:02.870622 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:02.870586 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" event={"ID":"7d6ef772-b5ae-413e-bed5-18b88335d224","Type":"ContainerStarted","Data":"07f67299a70214a44a934030386a4ff4c38d5f93675d9493508c361dfae88375"} Jul 09 10:24:04.859659 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:04.859628 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:24:05.545700 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:05.545663 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:24:05.546977 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:05.546950 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:24:05.588150 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:05.588091 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:24:05.588268 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:05.588195 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:24:05.588268 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:05.588239 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:24:05.895544 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:05.895501 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" event={"ID":"7d6ef772-b5ae-413e-bed5-18b88335d224","Type":"ContainerStarted","Data":"967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a"} Jul 09 10:24:05.920335 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:05.920293 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" podStartSLOduration=1.036384149 podStartE2EDuration="3.920278821s" totalImagesPullingTime="2.883894672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:24:05.91895219 +0000 UTC m=+1801.251435989" watchObservedRunningTime="2026-07-09 10:24:05.920278821 +0000 UTC m=+1801.252762619" Jul 09 10:24:06.548616 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:06.548583 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:06.553710 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:06.553685 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:06.901191 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:06.901155 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:06.902230 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:06.902209 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:20.990034 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:20.989998 2571 generic.go:356] "Generic (PLEG): container finished" podID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerID="967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a" exitCode=0 Jul 09 10:24:20.990523 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:20.990068 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" event={"ID":"7d6ef772-b5ae-413e-bed5-18b88335d224","Type":"ContainerDied","Data":"967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a"} Jul 09 10:24:20.990607 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:20.990590 2571 scope.go:117] "RemoveContainer" containerID="967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a" Jul 09 10:24:20.992563 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:20.992524 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:24:20.992641 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:20.992602 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:24:20.992641 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:20.992633 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:24:21.901724 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:21.901681 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:21.998978 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:21.998941 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" event={"ID":"7d6ef772-b5ae-413e-bed5-18b88335d224","Type":"ContainerStarted","Data":"1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6"} Jul 09 10:24:22.549766 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:22.549729 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:22.549967 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:22.549881 2571 patch_prober.go:28] interesting pod/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 09 10:24:22.549967 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:22.549928 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 09 10:24:23.548977 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:23.548928 2571 patch_prober.go:28] interesting pod/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 09 10:24:23.549506 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:23.549020 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 09 10:24:24.548835 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:24.548793 2571 patch_prober.go:28] interesting pod/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 09 10:24:24.549147 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:24.549096 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 09 10:24:25.548695 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:25.548646 2571 patch_prober.go:28] interesting pod/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 09 10:24:25.548913 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:25.548761 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 09 10:24:26.552524 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:26.552494 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:26.552955 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:26.552942 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:24:26.553586 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:24:26.553568 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:25:08.445555 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.445519 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp"] Jul 09 10:25:08.446271 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.445793 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" podUID="770a88d4-2ff4-4ecc-bcfa-f61229894172" containerName="maas-api" containerID="cri-o://7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6" gracePeriod=30 Jul 09 10:25:08.702788 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.702726 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:25:08.746793 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.746766 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dkmz\" (UniqueName: \"kubernetes.io/projected/770a88d4-2ff4-4ecc-bcfa-f61229894172-kube-api-access-9dkmz\") pod \"770a88d4-2ff4-4ecc-bcfa-f61229894172\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " Jul 09 10:25:08.746963 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.746800 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a88d4-2ff4-4ecc-bcfa-f61229894172-openshift-service-ca\") pod \"770a88d4-2ff4-4ecc-bcfa-f61229894172\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " Jul 09 10:25:08.746963 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.746848 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls\") pod \"770a88d4-2ff4-4ecc-bcfa-f61229894172\" (UID: \"770a88d4-2ff4-4ecc-bcfa-f61229894172\") " Jul 09 10:25:08.747235 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.747208 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770a88d4-2ff4-4ecc-bcfa-f61229894172-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "770a88d4-2ff4-4ecc-bcfa-f61229894172" (UID: "770a88d4-2ff4-4ecc-bcfa-f61229894172"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:25:08.748998 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.748967 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "770a88d4-2ff4-4ecc-bcfa-f61229894172" (UID: "770a88d4-2ff4-4ecc-bcfa-f61229894172"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 10:25:08.748998 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.748989 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770a88d4-2ff4-4ecc-bcfa-f61229894172-kube-api-access-9dkmz" (OuterVolumeSpecName: "kube-api-access-9dkmz") pod "770a88d4-2ff4-4ecc-bcfa-f61229894172" (UID: "770a88d4-2ff4-4ecc-bcfa-f61229894172"). InnerVolumeSpecName "kube-api-access-9dkmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:25:08.847530 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.847495 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dkmz\" (UniqueName: \"kubernetes.io/projected/770a88d4-2ff4-4ecc-bcfa-f61229894172-kube-api-access-9dkmz\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:25:08.847530 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.847527 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/770a88d4-2ff4-4ecc-bcfa-f61229894172-openshift-service-ca\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:25:08.847731 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:08.847541 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/770a88d4-2ff4-4ecc-bcfa-f61229894172-maas-api-tls\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:25:09.287725 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.287686 2571 generic.go:356] "Generic (PLEG): container finished" podID="770a88d4-2ff4-4ecc-bcfa-f61229894172" containerID="7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6" exitCode=0 Jul 09 10:25:09.287891 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.287736 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" Jul 09 10:25:09.287891 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.287728 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" event={"ID":"770a88d4-2ff4-4ecc-bcfa-f61229894172","Type":"ContainerDied","Data":"7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6"} Jul 09 10:25:09.287970 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.287903 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp" event={"ID":"770a88d4-2ff4-4ecc-bcfa-f61229894172","Type":"ContainerDied","Data":"e5d620681cd68dadb008d7dc831fb4d20e631c8be211908ef8cd4bbd2eddde1d"} Jul 09 10:25:09.287970 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.287936 2571 scope.go:117] "RemoveContainer" containerID="7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6" Jul 09 10:25:09.297386 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.297365 2571 scope.go:117] "RemoveContainer" containerID="7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6" Jul 09 10:25:09.297640 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:25:09.297624 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6\": container with ID starting with 7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6 not found: ID does not exist" containerID="7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6" Jul 09 10:25:09.297695 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.297649 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6"} err="failed to get container status \"7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6\": rpc error: code = NotFound desc = could not find container \"7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6\": container with ID starting with 7b886d2933a89351332f4536996073bb648f8e8dcfdc6372e9a94422f5f881c6 not found: ID does not exist" Jul 09 10:25:09.309676 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.309649 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp"] Jul 09 10:25:09.315872 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:09.315847 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-31eb62-586b568fdf-55gcp"] Jul 09 10:25:11.161048 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:11.161014 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="770a88d4-2ff4-4ecc-bcfa-f61229894172" path="/var/lib/kubelet/pods/770a88d4-2ff4-4ecc-bcfa-f61229894172/volumes" Jul 09 10:25:59.069911 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:59.069878 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5"] Jul 09 10:25:59.070492 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:25:59.070101 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" containerID="cri-o://1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6" gracePeriod=30 Jul 09 10:26:04.336961 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.336899 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:26:04.358457 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.358396 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-credential-socket\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.358626 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.358572 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-podinfo\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.358626 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.358618 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-token\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.358750 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.358656 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-data\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.358750 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.358681 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-certs\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.358855 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.358749 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:04.358987 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.358966 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:04.359277 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.359257 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-socket\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.359407 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.359344 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ef772-b5ae-413e-bed5-18b88335d224-istiod-ca-cert\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.359474 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.359422 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r66cr\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-kube-api-access-r66cr\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.359533 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.359470 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-envoy\") pod \"7d6ef772-b5ae-413e-bed5-18b88335d224\" (UID: \"7d6ef772-b5ae-413e-bed5-18b88335d224\") " Jul 09 10:26:04.359894 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.359760 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-credential-socket\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.359894 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.359783 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-certs\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.359894 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.359777 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:04.360109 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.359905 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6ef772-b5ae-413e-bed5-18b88335d224-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:26:04.360109 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.360041 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:04.361621 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.361576 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 10:26:04.361742 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.361613 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:26:04.362103 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.362083 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:04.362202 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.362155 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-kube-api-access-r66cr" (OuterVolumeSpecName: "kube-api-access-r66cr") pod "7d6ef772-b5ae-413e-bed5-18b88335d224" (UID: "7d6ef772-b5ae-413e-bed5-18b88335d224"). InnerVolumeSpecName "kube-api-access-r66cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:26:04.460733 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.460699 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-podinfo\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.460733 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.460727 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-token\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.460733 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.460737 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-data\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.460956 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.460750 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-workload-socket\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.460956 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.460762 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d6ef772-b5ae-413e-bed5-18b88335d224-istiod-ca-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.460956 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.460770 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r66cr\" (UniqueName: \"kubernetes.io/projected/7d6ef772-b5ae-413e-bed5-18b88335d224-kube-api-access-r66cr\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.460956 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.460781 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d6ef772-b5ae-413e-bed5-18b88335d224-istio-envoy\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:04.597191 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.597075 2571 generic.go:356] "Generic (PLEG): container finished" podID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerID="1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6" exitCode=0 Jul 09 10:26:04.597191 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.597148 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" Jul 09 10:26:04.597191 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.597160 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" event={"ID":"7d6ef772-b5ae-413e-bed5-18b88335d224","Type":"ContainerDied","Data":"1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6"} Jul 09 10:26:04.597423 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.597241 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5" event={"ID":"7d6ef772-b5ae-413e-bed5-18b88335d224","Type":"ContainerDied","Data":"07f67299a70214a44a934030386a4ff4c38d5f93675d9493508c361dfae88375"} Jul 09 10:26:04.597423 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.597265 2571 scope.go:117] "RemoveContainer" containerID="1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6" Jul 09 10:26:04.607554 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.607541 2571 scope.go:117] "RemoveContainer" containerID="967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a" Jul 09 10:26:04.616151 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.616120 2571 scope.go:117] "RemoveContainer" containerID="1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6" Jul 09 10:26:04.616423 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:26:04.616403 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6\": container with ID starting with 1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6 not found: ID does not exist" containerID="1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6" Jul 09 10:26:04.616474 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.616432 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6"} err="failed to get container status \"1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6\": rpc error: code = NotFound desc = could not find container \"1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6\": container with ID starting with 1626aef199cbb8e7c181680238795abaa9d5d422dbedabee8bc4d41826de32c6 not found: ID does not exist" Jul 09 10:26:04.616474 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.616450 2571 scope.go:117] "RemoveContainer" containerID="967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a" Jul 09 10:26:04.616683 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:26:04.616666 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a\": container with ID starting with 967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a not found: ID does not exist" containerID="967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a" Jul 09 10:26:04.616724 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.616690 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a"} err="failed to get container status \"967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a\": rpc error: code = NotFound desc = could not find container \"967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a\": container with ID starting with 967631d915669e211b94d529184d7e48f9d43e98ec00a53f639cfe778d943c6a not found: ID does not exist" Jul 09 10:26:04.624630 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.624605 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5"] Jul 09 10:26:04.628989 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:04.628964 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-14869d-openshift-default-c77974fff-f8ct5"] Jul 09 10:26:05.161793 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:05.161764 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" path="/var/lib/kubelet/pods/7d6ef772-b5ae-413e-bed5-18b88335d224/volumes" Jul 09 10:26:08.770140 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.770101 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f"] Jul 09 10:26:08.770567 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.770550 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" Jul 09 10:26:08.770626 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.770570 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" Jul 09 10:26:08.770626 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.770585 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="770a88d4-2ff4-4ecc-bcfa-f61229894172" containerName="maas-api" Jul 09 10:26:08.770626 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.770593 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="770a88d4-2ff4-4ecc-bcfa-f61229894172" containerName="maas-api" Jul 09 10:26:08.770725 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.770673 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" Jul 09 10:26:08.770725 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.770684 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="770a88d4-2ff4-4ecc-bcfa-f61229894172" containerName="maas-api" Jul 09 10:26:08.773825 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.773804 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.776723 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.776700 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 10:26:08.776858 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.776841 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2b323e-openshift-default-dockercfg-szmrq\"" Jul 09 10:26:08.785668 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.785647 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f"] Jul 09 10:26:08.905783 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.905750 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-podinfo\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.905954 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.905788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-certs\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.905954 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.905807 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7609c429-5e99-4ee7-8588-1f12481bbfb3-istiod-ca-cert\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.905954 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.905835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-data\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.905954 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.905897 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zcs\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-kube-api-access-m8zcs\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.905954 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.905952 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-envoy\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.906150 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.905982 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-credential-socket\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.906150 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.906011 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-token\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:08.906150 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:08.906081 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-socket\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007189 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007154 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-data\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007189 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007194 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zcs\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-kube-api-access-m8zcs\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007434 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-envoy\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007434 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-credential-socket\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007434 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007304 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-token\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007434 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-socket\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007434 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-podinfo\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007434 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-certs\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007766 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007441 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7609c429-5e99-4ee7-8588-1f12481bbfb3-istiod-ca-cert\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007766 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007706 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-credential-socket\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007766 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-socket\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.007766 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007760 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-certs\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.008009 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.007844 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-data\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.008114 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.008095 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7609c429-5e99-4ee7-8588-1f12481bbfb3-istiod-ca-cert\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.009505 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.009485 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-envoy\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.009820 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.009801 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-podinfo\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.028420 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.028355 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-token\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.030204 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.030179 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zcs\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-kube-api-access-m8zcs\") pod \"e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.085031 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.085001 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:09.230563 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.230536 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f"] Jul 09 10:26:09.232836 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:26:09.232806 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7609c429_5e99_4ee7_8588_1f12481bbfb3.slice/crio-969343ce75d7c0dd46ccb93270955c33c740ebede3bd7e8f932d8bb0a8a20fdd WatchSource:0}: Error finding container 969343ce75d7c0dd46ccb93270955c33c740ebede3bd7e8f932d8bb0a8a20fdd: Status 404 returned error can't find the container with id 969343ce75d7c0dd46ccb93270955c33c740ebede3bd7e8f932d8bb0a8a20fdd Jul 09 10:26:09.234832 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.234798 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:09.234913 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.234862 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:09.234913 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.234892 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:09.629625 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.629527 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" event={"ID":"7609c429-5e99-4ee7-8588-1f12481bbfb3","Type":"ContainerStarted","Data":"4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8"} Jul 09 10:26:09.629625 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.629589 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" event={"ID":"7609c429-5e99-4ee7-8588-1f12481bbfb3","Type":"ContainerStarted","Data":"969343ce75d7c0dd46ccb93270955c33c740ebede3bd7e8f932d8bb0a8a20fdd"} Jul 09 10:26:09.653610 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:09.653549 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" podStartSLOduration=1.653528442 podStartE2EDuration="1.653528442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:26:09.651455086 +0000 UTC m=+1924.983938885" watchObservedRunningTime="2026-07-09 10:26:09.653528442 +0000 UTC m=+1924.986012242" Jul 09 10:26:10.086147 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:10.086091 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:10.092384 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:10.092358 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:10.635375 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:10.635344 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:10.636484 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:10.636464 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:23.712864 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:23.712830 2571 generic.go:356] "Generic (PLEG): container finished" podID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerID="4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8" exitCode=0 Jul 09 10:26:23.713295 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:23.712902 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" event={"ID":"7609c429-5e99-4ee7-8588-1f12481bbfb3","Type":"ContainerDied","Data":"4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8"} Jul 09 10:26:23.713403 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:23.713383 2571 scope.go:117] "RemoveContainer" containerID="4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8" Jul 09 10:26:23.714450 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:23.714417 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:23.714551 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:23.714505 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:23.714589 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:23.714553 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:24.720543 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:24.720515 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" event={"ID":"7609c429-5e99-4ee7-8588-1f12481bbfb3","Type":"ContainerStarted","Data":"acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216"} Jul 09 10:26:25.085656 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:25.085621 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:25.086827 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:25.086794 2571 patch_prober.go:28] interesting pod/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 09 10:26:25.086962 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:25.086848 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 09 10:26:25.653290 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:25.653258 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:26.085874 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:26.085835 2571 patch_prober.go:28] interesting pod/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 09 10:26:26.086282 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:26.085910 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 09 10:26:27.085433 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:27.085395 2571 patch_prober.go:28] interesting pod/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 09 10:26:27.085610 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:27.085463 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 09 10:26:28.085743 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:28.085695 2571 patch_prober.go:28] interesting pod/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 09 10:26:28.086168 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:28.085776 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 09 10:26:29.089559 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:29.089530 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:29.090438 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:29.090418 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:39.848645 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:39.848613 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f"] Jul 09 10:26:39.849194 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:39.848834 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" containerID="cri-o://acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216" gracePeriod=30 Jul 09 10:26:40.770731 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.770694 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v"] Jul 09 10:26:40.771114 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.771102 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" Jul 09 10:26:40.771200 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.771115 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" Jul 09 10:26:40.771237 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.771212 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d6ef772-b5ae-413e-bed5-18b88335d224" containerName="istio-proxy" Jul 09 10:26:40.774426 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.774410 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.776926 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.776909 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-b5586e-openshift-default-dockercfg-psxjp\"" Jul 09 10:26:40.787477 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.787452 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v"] Jul 09 10:26:40.904983 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.904940 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrgz\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-kube-api-access-bcrgz\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.905452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.905024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-certs\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.905452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.905058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-envoy\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.905452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.905215 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-istio-token\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.905452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.905265 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f00df13-db08-442d-b13d-f85c9266aa32-istio-podinfo\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.905452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.905337 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-socket\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.905452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.905374 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-data\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.905452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.905412 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f00df13-db08-442d-b13d-f85c9266aa32-istiod-ca-cert\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:40.905452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:40.905439 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-credential-socket\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006108 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006058 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-istio-token\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006108 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006116 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f00df13-db08-442d-b13d-f85c9266aa32-istio-podinfo\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006168 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-socket\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006185 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-data\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006222 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f00df13-db08-442d-b13d-f85c9266aa32-istiod-ca-cert\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006259 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-credential-socket\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006383 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006317 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrgz\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-kube-api-access-bcrgz\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006657 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-certs\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006657 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006541 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-envoy\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006657 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006627 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-data\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006816 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006726 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-socket\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006816 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006790 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-credential-socket\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.006954 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.006933 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-certs\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.007118 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.007088 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f00df13-db08-442d-b13d-f85c9266aa32-istiod-ca-cert\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.008683 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.008661 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-envoy\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.008777 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.008665 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f00df13-db08-442d-b13d-f85c9266aa32-istio-podinfo\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.016989 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.016968 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-istio-token\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.017349 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.017330 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrgz\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-kube-api-access-bcrgz\") pod \"e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.085012 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.084979 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:41.438739 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.438711 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v"] Jul 09 10:26:41.440030 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:26:41.439997 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f00df13_db08_442d_b13d_f85c9266aa32.slice/crio-c925339c02f2297dfd1d2eed2de7bf8e34f283dd6013f93b42222c0856267aee WatchSource:0}: Error finding container c925339c02f2297dfd1d2eed2de7bf8e34f283dd6013f93b42222c0856267aee: Status 404 returned error can't find the container with id c925339c02f2297dfd1d2eed2de7bf8e34f283dd6013f93b42222c0856267aee Jul 09 10:26:41.442251 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.442219 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:41.442348 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.442286 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:41.442348 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.442314 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:41.832818 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.832785 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" event={"ID":"0f00df13-db08-442d-b13d-f85c9266aa32","Type":"ContainerStarted","Data":"b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28"} Jul 09 10:26:41.833011 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.832836 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" event={"ID":"0f00df13-db08-442d-b13d-f85c9266aa32","Type":"ContainerStarted","Data":"c925339c02f2297dfd1d2eed2de7bf8e34f283dd6013f93b42222c0856267aee"} Jul 09 10:26:41.855888 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:41.855843 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" podStartSLOduration=1.855827691 podStartE2EDuration="1.855827691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:26:41.854058677 +0000 UTC m=+1957.186542474" watchObservedRunningTime="2026-07-09 10:26:41.855827691 +0000 UTC m=+1957.188311489" Jul 09 10:26:42.085859 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:42.085784 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:42.090820 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:42.090792 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:42.839025 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:42.838986 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:42.839857 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:42.839837 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:26:44.092463 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:44.092418 2571 patch_prober.go:28] interesting pod/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 10:26:44.092833 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:44.092479 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 10:26:45.118514 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.118455 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:45.145729 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.145692 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8zcs\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-kube-api-access-m8zcs\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.145912 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.145749 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-certs\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.145912 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.145774 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-credential-socket\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.145912 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.145796 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-data\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.145912 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.145815 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-envoy\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.145912 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.145850 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-socket\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.146395 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.146367 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:45.146642 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.146614 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:45.146774 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.146709 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:45.147407 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.147379 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-data" (OuterVolumeSpecName: "istio-data") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:45.148727 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.148701 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:26:45.148829 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.148764 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-kube-api-access-m8zcs" (OuterVolumeSpecName: "kube-api-access-m8zcs") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "kube-api-access-m8zcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:26:45.246966 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.246933 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-podinfo\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.247162 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.246976 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-token\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.247162 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.247009 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7609c429-5e99-4ee7-8588-1f12481bbfb3-istiod-ca-cert\") pod \"7609c429-5e99-4ee7-8588-1f12481bbfb3\" (UID: \"7609c429-5e99-4ee7-8588-1f12481bbfb3\") " Jul 09 10:26:45.247294 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.247236 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-data\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.247294 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.247254 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-envoy\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.247294 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.247267 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-socket\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.247294 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.247279 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zcs\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-kube-api-access-m8zcs\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.247294 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.247294 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-workload-certs\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.247523 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.247307 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7609c429-5e99-4ee7-8588-1f12481bbfb3-credential-socket\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.247523 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.247477 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7609c429-5e99-4ee7-8588-1f12481bbfb3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:26:45.249055 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.249025 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 10:26:45.249228 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.249204 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-token" (OuterVolumeSpecName: "istio-token") pod "7609c429-5e99-4ee7-8588-1f12481bbfb3" (UID: "7609c429-5e99-4ee7-8588-1f12481bbfb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:26:45.348756 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.348726 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-podinfo\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.348756 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.348755 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7609c429-5e99-4ee7-8588-1f12481bbfb3-istio-token\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.348947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.348769 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7609c429-5e99-4ee7-8588-1f12481bbfb3-istiod-ca-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:26:45.856243 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.856206 2571 generic.go:356] "Generic (PLEG): container finished" podID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerID="acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216" exitCode=0 Jul 09 10:26:45.856414 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.856260 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" Jul 09 10:26:45.856414 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.856293 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" event={"ID":"7609c429-5e99-4ee7-8588-1f12481bbfb3","Type":"ContainerDied","Data":"acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216"} Jul 09 10:26:45.856414 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.856354 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f" event={"ID":"7609c429-5e99-4ee7-8588-1f12481bbfb3","Type":"ContainerDied","Data":"969343ce75d7c0dd46ccb93270955c33c740ebede3bd7e8f932d8bb0a8a20fdd"} Jul 09 10:26:45.856414 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.856378 2571 scope.go:117] "RemoveContainer" containerID="acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216" Jul 09 10:26:45.865688 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.865670 2571 scope.go:117] "RemoveContainer" containerID="4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8" Jul 09 10:26:45.874275 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.874259 2571 scope.go:117] "RemoveContainer" containerID="acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216" Jul 09 10:26:45.874532 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:26:45.874513 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216\": container with ID starting with acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216 not found: ID does not exist" containerID="acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216" Jul 09 10:26:45.874581 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.874539 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216"} err="failed to get container status \"acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216\": rpc error: code = NotFound desc = could not find container \"acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216\": container with ID starting with acb732b572d3ba0449e50d8583425d3fcc044330f46f614f982eed966bb01216 not found: ID does not exist" Jul 09 10:26:45.874581 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.874554 2571 scope.go:117] "RemoveContainer" containerID="4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8" Jul 09 10:26:45.874801 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:26:45.874786 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8\": container with ID starting with 4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8 not found: ID does not exist" containerID="4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8" Jul 09 10:26:45.874838 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.874807 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8"} err="failed to get container status \"4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8\": rpc error: code = NotFound desc = could not find container \"4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8\": container with ID starting with 4e3de26434bf8b8884d05f85a9c5d3f119dc3be8b9fca20a64b856c4366a30d8 not found: ID does not exist" Jul 09 10:26:45.887314 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.887288 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f"] Jul 09 10:26:45.889730 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:45.889699 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2b323e-openshift-default-6d4795c97f-2t47f"] Jul 09 10:26:47.160856 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:47.160820 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" path="/var/lib/kubelet/pods/7609c429-5e99-4ee7-8588-1f12481bbfb3/volumes" Jul 09 10:26:58.881298 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:58.881250 2571 patch_prober.go:28] interesting pod/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 10:26:58.881725 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:58.881320 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 10:26:59.936112 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:59.936029 2571 generic.go:356] "Generic (PLEG): container finished" podID="0f00df13-db08-442d-b13d-f85c9266aa32" containerID="b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28" exitCode=0 Jul 09 10:26:59.936498 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:59.936108 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" event={"ID":"0f00df13-db08-442d-b13d-f85c9266aa32","Type":"ContainerDied","Data":"b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28"} Jul 09 10:26:59.936541 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:59.936513 2571 scope.go:117] "RemoveContainer" containerID="b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28" Jul 09 10:26:59.937506 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:59.937480 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:59.937705 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:59.937553 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:26:59.937705 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:26:59.937582 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 09 10:27:00.949352 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:00.949280 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" event={"ID":"0f00df13-db08-442d-b13d-f85c9266aa32","Type":"ContainerStarted","Data":"6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7"} Jul 09 10:27:01.085218 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:01.085183 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:27:01.085412 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:01.085388 2571 patch_prober.go:28] interesting pod/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 09 10:27:01.085467 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:01.085438 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 09 10:27:02.085616 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:02.085579 2571 patch_prober.go:28] interesting pod/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 09 10:27:02.086012 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:02.085644 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 09 10:27:03.085745 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:03.085703 2571 patch_prober.go:28] interesting pod/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 09 10:27:03.086159 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:03.085773 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 09 10:27:04.085531 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:04.085486 2571 patch_prober.go:28] interesting pod/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Jul 09 10:27:04.085739 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:04.085586 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Jul 09 10:27:05.088582 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:05.088548 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:27:05.089040 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:05.089023 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:27:05.089668 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:27:05.089650 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:29:05.581542 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:29:05.581510 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:29:05.584820 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:29:05.584796 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:30:00.164880 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.164788 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns"] Jul 09 10:30:00.165385 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.165240 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" Jul 09 10:30:00.165385 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.165253 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" Jul 09 10:30:00.165385 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.165328 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" Jul 09 10:30:00.168452 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.168434 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" Jul 09 10:30:00.171776 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.171748 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-npsbz\"" Jul 09 10:30:00.173664 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.173635 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b"] Jul 09 10:30:00.174260 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.174240 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" Jul 09 10:30:00.174346 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.174261 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" Jul 09 10:30:00.174408 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.174383 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7609c429-5e99-4ee7-8588-1f12481bbfb3" containerName="istio-proxy" Jul 09 10:30:00.177680 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.177662 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8"] Jul 09 10:30:00.177835 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.177817 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" Jul 09 10:30:00.181117 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.181100 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" Jul 09 10:30:00.208325 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.208179 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns"] Jul 09 10:30:00.209740 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.209695 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b"] Jul 09 10:30:00.214263 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.214235 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8"] Jul 09 10:30:00.300931 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.300888 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw98f\" (UniqueName: \"kubernetes.io/projected/da41a577-9c02-4d9f-bafe-cd0c3f41ce1a-kube-api-access-qw98f\") pod \"maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns\" (UID: \"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" Jul 09 10:30:00.301120 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.300948 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wht4\" (UniqueName: \"kubernetes.io/projected/af1310ad-2662-4bb5-b827-0534f11bd8d9-kube-api-access-8wht4\") pod \"maas-api-key-cleanup-29726550-w5dg8\" (UID: \"af1310ad-2662-4bb5-b827-0534f11bd8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" Jul 09 10:30:00.301120 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.301032 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxjf\" (UniqueName: \"kubernetes.io/projected/b10beadf-9a15-4d3f-befa-b8c149e80f8e-kube-api-access-rgxjf\") pod \"maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b\" (UID: \"b10beadf-9a15-4d3f-befa-b8c149e80f8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" Jul 09 10:30:00.402420 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.402361 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw98f\" (UniqueName: \"kubernetes.io/projected/da41a577-9c02-4d9f-bafe-cd0c3f41ce1a-kube-api-access-qw98f\") pod \"maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns\" (UID: \"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" Jul 09 10:30:00.402420 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.402425 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wht4\" (UniqueName: \"kubernetes.io/projected/af1310ad-2662-4bb5-b827-0534f11bd8d9-kube-api-access-8wht4\") pod \"maas-api-key-cleanup-29726550-w5dg8\" (UID: \"af1310ad-2662-4bb5-b827-0534f11bd8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" Jul 09 10:30:00.402654 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.402456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxjf\" (UniqueName: \"kubernetes.io/projected/b10beadf-9a15-4d3f-befa-b8c149e80f8e-kube-api-access-rgxjf\") pod \"maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b\" (UID: \"b10beadf-9a15-4d3f-befa-b8c149e80f8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" Jul 09 10:30:00.416276 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.416207 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw98f\" (UniqueName: \"kubernetes.io/projected/da41a577-9c02-4d9f-bafe-cd0c3f41ce1a-kube-api-access-qw98f\") pod \"maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns\" (UID: \"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" Jul 09 10:30:00.416276 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.416207 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wht4\" (UniqueName: \"kubernetes.io/projected/af1310ad-2662-4bb5-b827-0534f11bd8d9-kube-api-access-8wht4\") pod \"maas-api-key-cleanup-29726550-w5dg8\" (UID: \"af1310ad-2662-4bb5-b827-0534f11bd8d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" Jul 09 10:30:00.416276 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.416265 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxjf\" (UniqueName: \"kubernetes.io/projected/b10beadf-9a15-4d3f-befa-b8c149e80f8e-kube-api-access-rgxjf\") pod \"maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b\" (UID: \"b10beadf-9a15-4d3f-befa-b8c149e80f8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" Jul 09 10:30:00.479367 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.479330 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" Jul 09 10:30:00.488561 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.488542 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" Jul 09 10:30:00.494454 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.494416 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" Jul 09 10:30:00.856569 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.856537 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns"] Jul 09 10:30:00.857855 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:30:00.857826 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda41a577_9c02_4d9f_bafe_cd0c3f41ce1a.slice/crio-08e0b5a2f571f73848a4faaae0428ec322146cae5321064ca2e503ea0c7693ac WatchSource:0}: Error finding container 08e0b5a2f571f73848a4faaae0428ec322146cae5321064ca2e503ea0c7693ac: Status 404 returned error can't find the container with id 08e0b5a2f571f73848a4faaae0428ec322146cae5321064ca2e503ea0c7693ac Jul 09 10:30:00.890783 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.890759 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8"] Jul 09 10:30:00.893310 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:30:00.893282 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1310ad_2662_4bb5_b827_0534f11bd8d9.slice/crio-005c5a8af42dde1c2e23d92e40473d2eb6df391a594ccd55fd2e67a43e599c4b WatchSource:0}: Error finding container 005c5a8af42dde1c2e23d92e40473d2eb6df391a594ccd55fd2e67a43e599c4b: Status 404 returned error can't find the container with id 005c5a8af42dde1c2e23d92e40473d2eb6df391a594ccd55fd2e67a43e599c4b Jul 09 10:30:00.899153 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.899106 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b"] Jul 09 10:30:00.899245 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:30:00.899199 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10beadf_9a15_4d3f_befa_b8c149e80f8e.slice/crio-08a322d63d5442cfd869bd0fb7341f51a685d65d7a12355c9ea3cd6f03d45f9d WatchSource:0}: Error finding container 08a322d63d5442cfd869bd0fb7341f51a685d65d7a12355c9ea3cd6f03d45f9d: Status 404 returned error can't find the container with id 08a322d63d5442cfd869bd0fb7341f51a685d65d7a12355c9ea3cd6f03d45f9d Jul 09 10:30:00.973512 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.973473 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" event={"ID":"b10beadf-9a15-4d3f-befa-b8c149e80f8e","Type":"ContainerStarted","Data":"08a322d63d5442cfd869bd0fb7341f51a685d65d7a12355c9ea3cd6f03d45f9d"} Jul 09 10:30:00.974649 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.974621 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" event={"ID":"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a","Type":"ContainerStarted","Data":"08e0b5a2f571f73848a4faaae0428ec322146cae5321064ca2e503ea0c7693ac"} Jul 09 10:30:00.975733 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:00.975712 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" event={"ID":"af1310ad-2662-4bb5-b827-0534f11bd8d9","Type":"ContainerStarted","Data":"005c5a8af42dde1c2e23d92e40473d2eb6df391a594ccd55fd2e67a43e599c4b"} Jul 09 10:30:01.983348 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:01.983316 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" event={"ID":"b10beadf-9a15-4d3f-befa-b8c149e80f8e","Type":"ContainerStarted","Data":"f2ea96978db624a01fd77981089392cc57051aec572531038029ac847b7aba6e"} Jul 09 10:30:01.984877 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:01.984847 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" event={"ID":"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a","Type":"ContainerStarted","Data":"e92e7fb358901694bc9139fdb3ec822dbdb43d3ea6cb42c46f52d47c31eb4af7"} Jul 09 10:30:01.986495 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:01.986471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" event={"ID":"af1310ad-2662-4bb5-b827-0534f11bd8d9","Type":"ContainerStarted","Data":"1587175dcee543aa93271ceedefdd81ccb919ecbffcfd7e32f1dfc5070bba5a9"} Jul 09 10:30:02.025731 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:02.025674 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" podStartSLOduration=2.025661045 podStartE2EDuration="2.025661045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:30:02.022972397 +0000 UTC m=+2157.355456195" watchObservedRunningTime="2026-07-09 10:30:02.025661045 +0000 UTC m=+2157.358144843" Jul 09 10:30:02.074877 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:02.074831 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" podStartSLOduration=2.074816235 podStartE2EDuration="2.074816235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:30:02.07327574 +0000 UTC m=+2157.405759521" watchObservedRunningTime="2026-07-09 10:30:02.074816235 +0000 UTC m=+2157.407300033" Jul 09 10:30:02.800497 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:02.800440 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" podStartSLOduration=2.8004233960000002 podStartE2EDuration="2.800423396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:30:02.139770093 +0000 UTC m=+2157.472253891" watchObservedRunningTime="2026-07-09 10:30:02.800423396 +0000 UTC m=+2158.132907193" Jul 09 10:30:02.820867 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:02.820831 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns"] Jul 09 10:30:04.006225 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:04.006184 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" podUID="da41a577-9c02-4d9f-bafe-cd0c3f41ce1a" containerName="cleanup" containerID="cri-o://e92e7fb358901694bc9139fdb3ec822dbdb43d3ea6cb42c46f52d47c31eb4af7" gracePeriod=30 Jul 09 10:30:08.649057 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:08.649023 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v"] Jul 09 10:30:08.649501 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:08.649275 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" containerID="cri-o://6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7" gracePeriod=30 Jul 09 10:30:10.066609 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:10.066574 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b"] Jul 09 10:30:10.067033 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:10.066786 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" podUID="b10beadf-9a15-4d3f-befa-b8c149e80f8e" containerName="cleanup" containerID="cri-o://f2ea96978db624a01fd77981089392cc57051aec572531038029ac847b7aba6e" gracePeriod=30 Jul 09 10:30:13.921058 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:13.920996 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:30:14.005531 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005501 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-envoy\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.005704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005539 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-data\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.005704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005560 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-socket\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.005704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005577 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f00df13-db08-442d-b13d-f85c9266aa32-istiod-ca-cert\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.005704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005644 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-credential-socket\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.005704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005664 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcrgz\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-kube-api-access-bcrgz\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.005704 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005697 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f00df13-db08-442d-b13d-f85c9266aa32-istio-podinfo\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.006058 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005748 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-certs\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.006058 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005799 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-istio-token\") pod \"0f00df13-db08-442d-b13d-f85c9266aa32\" (UID: \"0f00df13-db08-442d-b13d-f85c9266aa32\") " Jul 09 10:30:14.006058 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.005895 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:30:14.006251 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.006048 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:30:14.006251 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.006152 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:30:14.006251 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.006173 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f00df13-db08-442d-b13d-f85c9266aa32-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 10:30:14.006251 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.006236 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-socket\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:14.006251 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.006252 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-credential-socket\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:14.006516 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.006262 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-workload-certs\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:14.006650 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.006623 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:30:14.007901 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.007874 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 10:30:14.008118 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.008097 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:30:14.008265 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.008245 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-kube-api-access-bcrgz" (OuterVolumeSpecName: "kube-api-access-bcrgz") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "kube-api-access-bcrgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:30:14.008308 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.008246 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f00df13-db08-442d-b13d-f85c9266aa32-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f00df13-db08-442d-b13d-f85c9266aa32" (UID: "0f00df13-db08-442d-b13d-f85c9266aa32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 10:30:14.065854 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.065820 2571 generic.go:356] "Generic (PLEG): container finished" podID="0f00df13-db08-442d-b13d-f85c9266aa32" containerID="6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7" exitCode=0 Jul 09 10:30:14.065996 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.065876 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" Jul 09 10:30:14.065996 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.065903 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" event={"ID":"0f00df13-db08-442d-b13d-f85c9266aa32","Type":"ContainerDied","Data":"6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7"} Jul 09 10:30:14.065996 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.065992 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v" event={"ID":"0f00df13-db08-442d-b13d-f85c9266aa32","Type":"ContainerDied","Data":"c925339c02f2297dfd1d2eed2de7bf8e34f283dd6013f93b42222c0856267aee"} Jul 09 10:30:14.066119 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.066015 2571 scope.go:117] "RemoveContainer" containerID="6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7" Jul 09 10:30:14.075657 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.075641 2571 scope.go:117] "RemoveContainer" containerID="b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28" Jul 09 10:30:14.083657 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.083639 2571 scope.go:117] "RemoveContainer" containerID="6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7" Jul 09 10:30:14.083911 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:30:14.083891 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7\": container with ID starting with 6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7 not found: ID does not exist" containerID="6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7" Jul 09 10:30:14.083965 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.083926 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7"} err="failed to get container status \"6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7\": rpc error: code = NotFound desc = could not find container \"6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7\": container with ID starting with 6e6cff9408c82013a2eb2254a4604fe418d6ca760cc5034b33696e466366f0d7 not found: ID does not exist" Jul 09 10:30:14.083965 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.083944 2571 scope.go:117] "RemoveContainer" containerID="b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28" Jul 09 10:30:14.084201 ip-10-0-140-69 kubenswrapper[2571]: E0709 10:30:14.084171 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28\": container with ID starting with b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28 not found: ID does not exist" containerID="b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28" Jul 09 10:30:14.084263 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.084205 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28"} err="failed to get container status \"b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28\": rpc error: code = NotFound desc = could not find container \"b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28\": container with ID starting with b4b2226c508aa24fa29e5caaf6d0f1740988b0bd7f188470b6ee19613c4fef28 not found: ID does not exist" Jul 09 10:30:14.093360 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.093341 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v"] Jul 09 10:30:14.098657 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.098638 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-b5586e-openshift-default-6d46d5fd7f-l8d2v"] Jul 09 10:30:14.107465 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.107448 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrgz\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-kube-api-access-bcrgz\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:14.107539 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.107467 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f00df13-db08-442d-b13d-f85c9266aa32-istio-podinfo\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:14.107539 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.107477 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f00df13-db08-442d-b13d-f85c9266aa32-istio-token\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:14.107539 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.107486 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-envoy\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:14.107539 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.107493 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f00df13-db08-442d-b13d-f85c9266aa32-istio-data\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:14.107539 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:14.107500 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f00df13-db08-442d-b13d-f85c9266aa32-istiod-ca-cert\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:15.167910 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:15.167876 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" path="/var/lib/kubelet/pods/0f00df13-db08-442d-b13d-f85c9266aa32/volumes" Jul 09 10:30:22.375806 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:22.375761 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6lq29_7c522cf7-aa1e-4e1a-82fb-2571a842c8c5/manager/0.log" Jul 09 10:30:22.508696 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:22.508671 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-47n9p_bec5ef67-8c48-44d6-9103-d73e75fa3d82/manager/0.log" Jul 09 10:30:22.637087 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:22.637014 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84494cff7b-c997q_49dda3df-d2e4-40e4-9fe2-e96094e6d743/manager/0.log" Jul 09 10:30:22.888231 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:22.888157 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-krxns_2558c0ba-ad0a-4c79-9ec3-01d4a99130f2/manager/0.log" Jul 09 10:30:23.288284 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:23.288208 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cb598dddb-lccbp_a70bdda0-c27f-4934-912a-9e2a4de490bb/copy-manifests/0.log" Jul 09 10:30:23.303874 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:23.303843 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cb598dddb-lccbp_a70bdda0-c27f-4934-912a-9e2a4de490bb/manager/0.log" Jul 09 10:30:24.164935 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.164908 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df_c1c8a9fb-f58f-4093-bbbf-174e83759d0c/pull/0.log" Jul 09 10:30:24.181919 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.181894 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df_c1c8a9fb-f58f-4093-bbbf-174e83759d0c/extract/0.log" Jul 09 10:30:24.194208 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.194185 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df_c1c8a9fb-f58f-4093-bbbf-174e83759d0c/util/0.log" Jul 09 10:30:24.317264 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.317234 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp_8ed4b1e8-5e49-4f14-9bf6-3e83d087c323/util/0.log" Jul 09 10:30:24.325410 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.325386 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp_8ed4b1e8-5e49-4f14-9bf6-3e83d087c323/pull/0.log" Jul 09 10:30:24.333212 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.333193 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp_8ed4b1e8-5e49-4f14-9bf6-3e83d087c323/extract/0.log" Jul 09 10:30:24.454175 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.454043 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh_603117d7-d538-4e54-8f7f-ace4639e908d/util/0.log" Jul 09 10:30:24.461053 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.461015 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh_603117d7-d538-4e54-8f7f-ace4639e908d/pull/0.log" Jul 09 10:30:24.469051 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.469034 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh_603117d7-d538-4e54-8f7f-ace4639e908d/extract/0.log" Jul 09 10:30:24.599759 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.599726 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5_99254337-7c21-4570-a5c7-d2f3ea3598ec/util/0.log" Jul 09 10:30:24.608436 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.608408 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5_99254337-7c21-4570-a5c7-d2f3ea3598ec/pull/0.log" Jul 09 10:30:24.616017 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.615998 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5_99254337-7c21-4570-a5c7-d2f3ea3598ec/extract/0.log" Jul 09 10:30:24.866267 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.866234 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vbfff_a11a241d-97f2-4ce8-af49-a556a3915b2f/manager/0.log" Jul 09 10:30:24.988467 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:24.988440 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-jx59n_2bcae414-b097-45ff-8668-108bafb4a6b9/manager/0.log" Jul 09 10:30:25.235770 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:25.235668 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tqztm_0bbecb6e-e078-40c6-95c0-78b28144ce0d/registry-server/0.log" Jul 09 10:30:25.365976 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:25.365946 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8z96b_db57175f-54e5-49ee-b451-130dbab8b490/manager/0.log" Jul 09 10:30:25.611904 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:25.611876 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4g2mv_719c52a8-0b0a-44c8-a150-bb5b4d5422e8/manager/0.log" Jul 09 10:30:26.359241 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:26.359213 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-589b555f76-zvd84_c0bea99a-6583-42ac-b649-d187076552e1/kube-auth-proxy/0.log" Jul 09 10:30:26.627331 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:26.627248 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-q5nz8_53671dad-c918-4117-8cee-a48b4a181ed8/payload-pre-processing/0.log" Jul 09 10:30:26.759048 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:26.759020 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xwmc9_96d11b03-2e8c-4a47-b533-fbd3630ab03f/payload-processing/0.log" Jul 09 10:30:26.884589 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:26.884491 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f6dfc6c5-m7gq8_ed295098-19e7-483e-88c5-20da6182ea79/router/0.log" Jul 09 10:30:27.247469 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:27.247379 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-sgsnr_5b4cb26b-8238-4a95-9fa1-a343900369e1/main/0.log" Jul 09 10:30:27.496009 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:27.495972 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-lnqx4_836f8679-f5bd-468c-a6e9-f20abf391d9b/main/0.log" Jul 09 10:30:32.010454 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.010418 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qjnrh/must-gather-j76pw"] Jul 09 10:30:32.011063 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.011045 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" Jul 09 10:30:32.011112 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.011067 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" Jul 09 10:30:32.011112 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.011085 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" Jul 09 10:30:32.011112 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.011094 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" Jul 09 10:30:32.011244 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.011190 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" Jul 09 10:30:32.011244 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.011200 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f00df13-db08-442d-b13d-f85c9266aa32" containerName="istio-proxy" Jul 09 10:30:32.014631 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.014616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qjnrh/must-gather-j76pw" Jul 09 10:30:32.017300 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.017271 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qjnrh\"/\"default-dockercfg-q7c7q\"" Jul 09 10:30:32.017448 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.017276 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qjnrh\"/\"openshift-service-ca.crt\"" Jul 09 10:30:32.017572 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.017556 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qjnrh\"/\"kube-root-ca.crt\"" Jul 09 10:30:32.028995 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.028973 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qjnrh/must-gather-j76pw"] Jul 09 10:30:32.068406 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.068372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bc294116-69f2-42e1-ada4-e6f181128e4b-must-gather-output\") pod \"must-gather-j76pw\" (UID: \"bc294116-69f2-42e1-ada4-e6f181128e4b\") " pod="openshift-must-gather-qjnrh/must-gather-j76pw" Jul 09 10:30:32.068567 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.068549 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqsh\" (UniqueName: \"kubernetes.io/projected/bc294116-69f2-42e1-ada4-e6f181128e4b-kube-api-access-tzqsh\") pod \"must-gather-j76pw\" (UID: \"bc294116-69f2-42e1-ada4-e6f181128e4b\") " pod="openshift-must-gather-qjnrh/must-gather-j76pw" Jul 09 10:30:32.169771 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.169737 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqsh\" (UniqueName: \"kubernetes.io/projected/bc294116-69f2-42e1-ada4-e6f181128e4b-kube-api-access-tzqsh\") pod \"must-gather-j76pw\" (UID: \"bc294116-69f2-42e1-ada4-e6f181128e4b\") " pod="openshift-must-gather-qjnrh/must-gather-j76pw" Jul 09 10:30:32.169771 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.169777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bc294116-69f2-42e1-ada4-e6f181128e4b-must-gather-output\") pod \"must-gather-j76pw\" (UID: \"bc294116-69f2-42e1-ada4-e6f181128e4b\") " pod="openshift-must-gather-qjnrh/must-gather-j76pw" Jul 09 10:30:32.170101 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.170083 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bc294116-69f2-42e1-ada4-e6f181128e4b-must-gather-output\") pod \"must-gather-j76pw\" (UID: \"bc294116-69f2-42e1-ada4-e6f181128e4b\") " pod="openshift-must-gather-qjnrh/must-gather-j76pw" Jul 09 10:30:32.183990 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.183965 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqsh\" (UniqueName: \"kubernetes.io/projected/bc294116-69f2-42e1-ada4-e6f181128e4b-kube-api-access-tzqsh\") pod \"must-gather-j76pw\" (UID: \"bc294116-69f2-42e1-ada4-e6f181128e4b\") " pod="openshift-must-gather-qjnrh/must-gather-j76pw" Jul 09 10:30:32.324220 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.324192 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qjnrh/must-gather-j76pw" Jul 09 10:30:32.488732 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.488705 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qjnrh/must-gather-j76pw"] Jul 09 10:30:32.490468 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:30:32.490439 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc294116_69f2_42e1_ada4_e6f181128e4b.slice/crio-7b34f332663ec495366444f105878174c04cc84ea84771199fdb34a129d67aa2 WatchSource:0}: Error finding container 7b34f332663ec495366444f105878174c04cc84ea84771199fdb34a129d67aa2: Status 404 returned error can't find the container with id 7b34f332663ec495366444f105878174c04cc84ea84771199fdb34a129d67aa2 Jul 09 10:30:32.492485 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:32.492465 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 10:30:33.174029 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:33.173967 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjnrh/must-gather-j76pw" event={"ID":"bc294116-69f2-42e1-ada4-e6f181128e4b","Type":"ContainerStarted","Data":"7b34f332663ec495366444f105878174c04cc84ea84771199fdb34a129d67aa2"} Jul 09 10:30:34.182690 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:34.182659 2571 generic.go:356] "Generic (PLEG): container finished" podID="da41a577-9c02-4d9f-bafe-cd0c3f41ce1a" containerID="e92e7fb358901694bc9139fdb3ec822dbdb43d3ea6cb42c46f52d47c31eb4af7" exitCode=137 Jul 09 10:30:34.183097 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:34.182701 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" event={"ID":"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a","Type":"ContainerDied","Data":"e92e7fb358901694bc9139fdb3ec822dbdb43d3ea6cb42c46f52d47c31eb4af7"} Jul 09 10:30:34.226875 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:34.226849 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" Jul 09 10:30:34.287304 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:34.287268 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw98f\" (UniqueName: \"kubernetes.io/projected/da41a577-9c02-4d9f-bafe-cd0c3f41ce1a-kube-api-access-qw98f\") pod \"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a\" (UID: \"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a\") " Jul 09 10:30:34.289940 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:34.289914 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da41a577-9c02-4d9f-bafe-cd0c3f41ce1a-kube-api-access-qw98f" (OuterVolumeSpecName: "kube-api-access-qw98f") pod "da41a577-9c02-4d9f-bafe-cd0c3f41ce1a" (UID: "da41a577-9c02-4d9f-bafe-cd0c3f41ce1a"). InnerVolumeSpecName "kube-api-access-qw98f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:30:34.388399 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:34.388361 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw98f\" (UniqueName: \"kubernetes.io/projected/da41a577-9c02-4d9f-bafe-cd0c3f41ce1a-kube-api-access-qw98f\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:35.193149 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:35.193094 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjnrh/must-gather-j76pw" event={"ID":"bc294116-69f2-42e1-ada4-e6f181128e4b","Type":"ContainerStarted","Data":"11337d4ca28be46aeebd4773e209259dff5808d28534319ff3846d001553d9ed"} Jul 09 10:30:35.193592 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:35.193155 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjnrh/must-gather-j76pw" event={"ID":"bc294116-69f2-42e1-ada4-e6f181128e4b","Type":"ContainerStarted","Data":"1146f83184fa600b2d78f2bd85b10c5cf54bbfa2c6544a846bc40f04151c5a14"} Jul 09 10:30:35.194428 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:35.194405 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" Jul 09 10:30:35.194428 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:35.194414 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns" event={"ID":"da41a577-9c02-4d9f-bafe-cd0c3f41ce1a","Type":"ContainerDied","Data":"08e0b5a2f571f73848a4faaae0428ec322146cae5321064ca2e503ea0c7693ac"} Jul 09 10:30:35.194633 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:35.194480 2571 scope.go:117] "RemoveContainer" containerID="e92e7fb358901694bc9139fdb3ec822dbdb43d3ea6cb42c46f52d47c31eb4af7" Jul 09 10:30:35.213372 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:35.213325 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qjnrh/must-gather-j76pw" podStartSLOduration=2.447951861 podStartE2EDuration="4.213307424s" totalImagesPullingTime="1.765355563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:30:35.21133893 +0000 UTC m=+2190.543822728" watchObservedRunningTime="2026-07-09 10:30:35.213307424 +0000 UTC m=+2190.545791223" Jul 09 10:30:35.234225 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:35.234183 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns"] Jul 09 10:30:35.239991 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:35.239952 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b5586e-29726550-x2vns"] Jul 09 10:30:36.097850 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:36.097821 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-slmhl_048fd61f-ba5b-41bc-95a5-366a5734571d/global-pull-secret-syncer/0.log" Jul 09 10:30:36.188487 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:36.188451 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tkpgz_b5181675-9c76-4ae5-bb57-dd4c8558b80f/konnectivity-agent/0.log" Jul 09 10:30:36.319296 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:36.319269 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-69.ec2.internal_f5d7380ed3c9d0d9e6311c53297db640/haproxy/0.log" Jul 09 10:30:37.163789 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:37.163756 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da41a577-9c02-4d9f-bafe-cd0c3f41ce1a" path="/var/lib/kubelet/pods/da41a577-9c02-4d9f-bafe-cd0c3f41ce1a/volumes" Jul 09 10:30:40.241639 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:40.241112 2571 generic.go:356] "Generic (PLEG): container finished" podID="b10beadf-9a15-4d3f-befa-b8c149e80f8e" containerID="f2ea96978db624a01fd77981089392cc57051aec572531038029ac847b7aba6e" exitCode=137 Jul 09 10:30:40.241639 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:40.241306 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" event={"ID":"b10beadf-9a15-4d3f-befa-b8c149e80f8e","Type":"ContainerDied","Data":"f2ea96978db624a01fd77981089392cc57051aec572531038029ac847b7aba6e"} Jul 09 10:30:40.437151 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:40.434832 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" Jul 09 10:30:40.557149 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:40.556473 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgxjf\" (UniqueName: \"kubernetes.io/projected/b10beadf-9a15-4d3f-befa-b8c149e80f8e-kube-api-access-rgxjf\") pod \"b10beadf-9a15-4d3f-befa-b8c149e80f8e\" (UID: \"b10beadf-9a15-4d3f-befa-b8c149e80f8e\") " Jul 09 10:30:40.568153 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:40.566245 2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10beadf-9a15-4d3f-befa-b8c149e80f8e-kube-api-access-rgxjf" (OuterVolumeSpecName: "kube-api-access-rgxjf") pod "b10beadf-9a15-4d3f-befa-b8c149e80f8e" (UID: "b10beadf-9a15-4d3f-befa-b8c149e80f8e"). InnerVolumeSpecName "kube-api-access-rgxjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 10:30:40.657522 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:40.657429 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgxjf\" (UniqueName: \"kubernetes.io/projected/b10beadf-9a15-4d3f-befa-b8c149e80f8e-kube-api-access-rgxjf\") on node \"ip-10-0-140-69.ec2.internal\" DevicePath \"\"" Jul 09 10:30:41.120996 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.120953 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df_c1c8a9fb-f58f-4093-bbbf-174e83759d0c/extract/0.log" Jul 09 10:30:41.159070 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.159039 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df_c1c8a9fb-f58f-4093-bbbf-174e83759d0c/util/0.log" Jul 09 10:30:41.208719 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.208684 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759jk8df_c1c8a9fb-f58f-4093-bbbf-174e83759d0c/pull/0.log" Jul 09 10:30:41.252081 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.252035 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" event={"ID":"b10beadf-9a15-4d3f-befa-b8c149e80f8e","Type":"ContainerDied","Data":"08a322d63d5442cfd869bd0fb7341f51a685d65d7a12355c9ea3cd6f03d45f9d"} Jul 09 10:30:41.252627 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.252143 2571 scope.go:117] "RemoveContainer" containerID="f2ea96978db624a01fd77981089392cc57051aec572531038029ac847b7aba6e" Jul 09 10:30:41.252627 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.252326 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b" Jul 09 10:30:41.255462 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.255412 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp_8ed4b1e8-5e49-4f14-9bf6-3e83d087c323/extract/0.log" Jul 09 10:30:41.286323 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.286286 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b"] Jul 09 10:30:41.298145 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.297629 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-a0b44e-29726550-9tz5b"] Jul 09 10:30:41.298495 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.298468 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp_8ed4b1e8-5e49-4f14-9bf6-3e83d087c323/util/0.log" Jul 09 10:30:41.339495 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.339457 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e082tbp_8ed4b1e8-5e49-4f14-9bf6-3e83d087c323/pull/0.log" Jul 09 10:30:41.412086 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.412006 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh_603117d7-d538-4e54-8f7f-ace4639e908d/extract/0.log" Jul 09 10:30:41.464402 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.464377 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh_603117d7-d538-4e54-8f7f-ace4639e908d/util/0.log" Jul 09 10:30:41.536044 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.536016 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fm7zh_603117d7-d538-4e54-8f7f-ace4639e908d/pull/0.log" Jul 09 10:30:41.595209 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.595180 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5_99254337-7c21-4570-a5c7-d2f3ea3598ec/extract/0.log" Jul 09 10:30:41.632677 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.632653 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5_99254337-7c21-4570-a5c7-d2f3ea3598ec/util/0.log" Jul 09 10:30:41.681880 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:41.681797 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1vd5d5_99254337-7c21-4570-a5c7-d2f3ea3598ec/pull/0.log" Jul 09 10:30:42.205505 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:42.205464 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vbfff_a11a241d-97f2-4ce8-af49-a556a3915b2f/manager/0.log" Jul 09 10:30:42.250509 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:42.250472 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-jx59n_2bcae414-b097-45ff-8668-108bafb4a6b9/manager/0.log" Jul 09 10:30:42.356247 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:42.356210 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tqztm_0bbecb6e-e078-40c6-95c0-78b28144ce0d/registry-server/0.log" Jul 09 10:30:42.461198 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:42.460696 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8z96b_db57175f-54e5-49ee-b451-130dbab8b490/manager/0.log" Jul 09 10:30:42.593548 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:42.593517 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4g2mv_719c52a8-0b0a-44c8-a150-bb5b4d5422e8/manager/0.log" Jul 09 10:30:43.169469 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:43.169431 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b10beadf-9a15-4d3f-befa-b8c149e80f8e" path="/var/lib/kubelet/pods/b10beadf-9a15-4d3f-befa-b8c149e80f8e/volumes" Jul 09 10:30:44.643369 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:44.643316 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-q7kml_35d1d4bd-3fef-4335-a2b3-8085386414da/cluster-monitoring-operator/0.log" Jul 09 10:30:45.160853 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:45.160825 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4dq8_5781a5e1-10e2-421d-9a75-9be4dfa352e5/node-exporter/0.log" Jul 09 10:30:45.191321 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:45.191294 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4dq8_5781a5e1-10e2-421d-9a75-9be4dfa352e5/kube-rbac-proxy/0.log" Jul 09 10:30:45.227433 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:45.227403 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4dq8_5781a5e1-10e2-421d-9a75-9be4dfa352e5/init-textfile/0.log" Jul 09 10:30:46.314468 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.314434 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95"] Jul 09 10:30:46.314947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.314873 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b10beadf-9a15-4d3f-befa-b8c149e80f8e" containerName="cleanup" Jul 09 10:30:46.314947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.314888 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b10beadf-9a15-4d3f-befa-b8c149e80f8e" containerName="cleanup" Jul 09 10:30:46.314947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.314904 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da41a577-9c02-4d9f-bafe-cd0c3f41ce1a" containerName="cleanup" Jul 09 10:30:46.314947 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.314910 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="da41a577-9c02-4d9f-bafe-cd0c3f41ce1a" containerName="cleanup" Jul 09 10:30:46.315197 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.314976 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b10beadf-9a15-4d3f-befa-b8c149e80f8e" containerName="cleanup" Jul 09 10:30:46.315197 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.314987 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="da41a577-9c02-4d9f-bafe-cd0c3f41ce1a" containerName="cleanup" Jul 09 10:30:46.322155 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.322105 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.332530 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.332506 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95"] Jul 09 10:30:46.428268 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.428232 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-sys\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.428453 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.428296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wszv\" (UniqueName: \"kubernetes.io/projected/2dcef927-ed87-4d9c-9898-1d5f99970fcb-kube-api-access-8wszv\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.428453 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.428416 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-proc\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.428560 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.428458 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-lib-modules\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.428560 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.428500 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-podres\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.528955 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.528919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wszv\" (UniqueName: \"kubernetes.io/projected/2dcef927-ed87-4d9c-9898-1d5f99970fcb-kube-api-access-8wszv\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.529160 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.529068 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-proc\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.529160 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.529090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-lib-modules\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.529160 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.529113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-podres\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.529343 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.529184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-sys\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.529343 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.529223 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-proc\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.529343 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.529289 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-lib-modules\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.529487 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.529342 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-podres\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.529487 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.529347 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dcef927-ed87-4d9c-9898-1d5f99970fcb-sys\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.539500 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.539470 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wszv\" (UniqueName: \"kubernetes.io/projected/2dcef927-ed87-4d9c-9898-1d5f99970fcb-kube-api-access-8wszv\") pod \"perf-node-gather-daemonset-wgd95\" (UID: \"2dcef927-ed87-4d9c-9898-1d5f99970fcb\") " pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.635836 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.635728 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:46.814541 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:46.814518 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95"] Jul 09 10:30:46.816952 ip-10-0-140-69 kubenswrapper[2571]: W0709 10:30:46.816920 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2dcef927_ed87_4d9c_9898_1d5f99970fcb.slice/crio-8f014c8f6974f64065b737d200160ba9cb9e727bc19a08c5d387371cf63b3803 WatchSource:0}: Error finding container 8f014c8f6974f64065b737d200160ba9cb9e727bc19a08c5d387371cf63b3803: Status 404 returned error can't find the container with id 8f014c8f6974f64065b737d200160ba9cb9e727bc19a08c5d387371cf63b3803 Jul 09 10:30:47.320619 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:47.320585 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" event={"ID":"2dcef927-ed87-4d9c-9898-1d5f99970fcb","Type":"ContainerStarted","Data":"5c6a70891ec9d41f276355d83aa478e7d3863f9ee76b083c39b126d27393e5c6"} Jul 09 10:30:47.321079 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:47.320644 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" event={"ID":"2dcef927-ed87-4d9c-9898-1d5f99970fcb","Type":"ContainerStarted","Data":"8f014c8f6974f64065b737d200160ba9cb9e727bc19a08c5d387371cf63b3803"} Jul 09 10:30:47.321079 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:47.320670 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:47.350439 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:47.350385 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" podStartSLOduration=1.35036905 podStartE2EDuration="1.35036905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:30:47.346660322 +0000 UTC m=+2202.679144121" watchObservedRunningTime="2026-07-09 10:30:47.35036905 +0000 UTC m=+2202.682852848" Jul 09 10:30:47.783058 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:47.782973 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/2.log" Jul 09 10:30:47.787196 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:47.787171 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-xph5s_48bad99c-20b4-4c34-b8eb-67d0e7189c8c/console-operator/3.log" Jul 09 10:30:48.281565 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:48.281539 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66d476967d-lk75b_07e20594-dc24-4ddd-a6dc-980054710224/console/0.log" Jul 09 10:30:48.931141 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:48.931093 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-fb498_bee94409-6c0f-47f2-8107-f0e93537ce33/volume-data-source-validator/0.log" Jul 09 10:30:50.126787 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:50.126759 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wp9sj_5b882c2a-8f11-4f25-aeb0-da5a38087884/dns/0.log" Jul 09 10:30:50.157527 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:50.157501 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wp9sj_5b882c2a-8f11-4f25-aeb0-da5a38087884/kube-rbac-proxy/0.log" Jul 09 10:30:50.296636 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:50.296612 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k57ns_8ff1e23f-f24b-4568-b9ee-1f0692a9afe3/dns-node-resolver/0.log" Jul 09 10:30:51.110810 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:51.110779 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l84jq_d31879ac-51f8-4f04-8591-dd2c0487c68b/node-ca/0.log" Jul 09 10:30:52.349406 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:52.349379 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-589b555f76-zvd84_c0bea99a-6583-42ac-b649-d187076552e1/kube-auth-proxy/0.log" Jul 09 10:30:52.462670 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:52.462639 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-q5nz8_53671dad-c918-4117-8cee-a48b4a181ed8/payload-pre-processing/0.log" Jul 09 10:30:52.519306 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:52.519269 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xwmc9_96d11b03-2e8c-4a47-b533-fbd3630ab03f/payload-processing/0.log" Jul 09 10:30:52.546145 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:52.546106 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f6dfc6c5-m7gq8_ed295098-19e7-483e-88c5-20da6182ea79/router/0.log" Jul 09 10:30:53.328208 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:53.328181 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6zbgj_de11cbd5-8899-4f7f-b5db-e2236850cb4c/serve-healthcheck-canary/0.log" Jul 09 10:30:53.337434 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:53.337408 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qjnrh/perf-node-gather-daemonset-wgd95" Jul 09 10:30:53.980165 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:53.980111 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-fpvm6_90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4/insights-operator/0.log" Jul 09 10:30:53.982331 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:53.982309 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-fpvm6_90c650b7-c9b1-43a5-b6a4-4c0a9d68f7b4/insights-operator/1.log" Jul 09 10:30:54.113542 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:54.113511 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k2n6k_ffe1512c-1baa-4b07-996b-38dcd28ebcec/kube-rbac-proxy/0.log" Jul 09 10:30:54.148464 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:54.148438 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k2n6k_ffe1512c-1baa-4b07-996b-38dcd28ebcec/exporter/0.log" Jul 09 10:30:54.184846 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:54.184817 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k2n6k_ffe1512c-1baa-4b07-996b-38dcd28ebcec/extractor/0.log" Jul 09 10:30:56.508430 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:56.508398 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-589b555f76-zvd84_c0bea99a-6583-42ac-b649-d187076552e1/kube-auth-proxy/0.log" Jul 09 10:30:56.626528 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:56.626495 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-q5nz8_53671dad-c918-4117-8cee-a48b4a181ed8/payload-pre-processing/0.log" Jul 09 10:30:56.709950 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:56.709917 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xwmc9_96d11b03-2e8c-4a47-b533-fbd3630ab03f/payload-processing/0.log" Jul 09 10:30:56.762758 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:56.762729 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f6dfc6c5-m7gq8_ed295098-19e7-483e-88c5-20da6182ea79/router/0.log" Jul 09 10:30:56.877906 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:56.877876 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-6lq29_7c522cf7-aa1e-4e1a-82fb-2571a842c8c5/manager/0.log" Jul 09 10:30:57.024119 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:57.024089 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-47n9p_bec5ef67-8c48-44d6-9103-d73e75fa3d82/manager/0.log" Jul 09 10:30:57.134772 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:57.134653 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84494cff7b-c997q_49dda3df-d2e4-40e4-9fe2-e96094e6d743/manager/0.log" Jul 09 10:30:57.266939 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:57.266881 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-krxns_2558c0ba-ad0a-4c79-9ec3-01d4a99130f2/manager/0.log" Jul 09 10:30:57.533975 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:57.533864 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cb598dddb-lccbp_a70bdda0-c27f-4934-912a-9e2a4de490bb/manager/0.log" Jul 09 10:30:57.589649 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:57.589619 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cb598dddb-lccbp_a70bdda0-c27f-4934-912a-9e2a4de490bb/copy-manifests/0.log" Jul 09 10:30:58.265246 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:58.265171 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-589b555f76-zvd84_c0bea99a-6583-42ac-b649-d187076552e1/kube-auth-proxy/0.log" Jul 09 10:30:58.366309 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:58.366272 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-q5nz8_53671dad-c918-4117-8cee-a48b4a181ed8/payload-pre-processing/0.log" Jul 09 10:30:58.413342 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:58.413307 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xwmc9_96d11b03-2e8c-4a47-b533-fbd3630ab03f/payload-processing/0.log" Jul 09 10:30:58.432036 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:30:58.432008 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56f6dfc6c5-m7gq8_ed295098-19e7-483e-88c5-20da6182ea79/router/0.log" Jul 09 10:31:00.013597 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:31:00.013559 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8"] Jul 09 10:31:00.014015 ip-10-0-140-69 kubenswrapper[2571]: I0709 10:31:00.013802 2571 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726550-w5dg8" podUID="af1310ad-2662-4bb5-b827-0534f11bd8d9" containerName="cleanup" containerID="cri-o://1587175dcee543aa93271ceedefdd81ccb919ecbffcfd7e32f1dfc5070bba5a9" gracePeriod=30